IO500 - Submissions

Oakforest-PACS

INFORMATION

System	Oakforest-PACS
Storage Vendor	DDN
Filesystem Type	IME
Filesystem Name
Filesystem Version	1.2.2

Institution	JCAHPC
Client Procs Per Node
Client Operating System	CentOS
Client Operating System Version
Client Kernel Version

METADATA SERVER

Storage Type	SSD
Volatile Memory
Storage Interface
Network	OmniPath
Software Version
OS Version

DATA SERVER

Storage Type	SSD
Volatile Memory
Storage Interface	NVMe
Network	OmniPath
Software Version
OS Version

IO500 SCORES

IO500 Score	168.09
IO500 BW	581.43 GiB/s
IO500 MD	48.60 kIOP/s

INFORMATION

Client Nodes	2,048
Client Total Procs	16,384
Metadata Nodes	1
Metadata Storage Devices	12
Data Nodes	25
Data Storage Devices	1,200

IOR & FIND

Easy Write	917.79 GiB/s
Easy Read	1,000.35 GiB/s
Hard Write	402.19 GiB/s
Hard Read	309.50 GiB/s
Find	272.00 kIOP/s

METADATA

Easy Write	51.39 kIOP/s
Easy Stat	180.01 kIOP/s
Easy Delete	38.92 kIOP/s
Hard Write	13.09 kIOP/s
Hard Read	126.68 kIOP/s
Hard Stat	40.47 kIOP/s
Hard Delete	4.74 kIOP/s

Submitted Files

ior_easy_read

IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Nov 14 08:08:36 2019
Command line        : /cache/xg17i000/x10007/io500-sc-19/bin/ior -r -R -t 2048k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy/ior_file_easy -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy/stonewall
Machine             : Linux c0468.ofp
TestID              : 0
StartTime           : Thu Nov 14 08:08:36 2019
Path                : /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy
FS                  : 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.9%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 16384
clients per node    : 8
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 155000 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 170424302305280000.
WARNING: Stat() of aggregate file size      = 330300164931584.
WARNING: Using actual aggregate bytes moved = 330300164931584.
read      1024360    10158080000 2048.00    0.417171   307.08     0.018064   307.51     0   
Max Read:  1024359.80 MiB/sec (1074119.10 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read      1024359.80 1024359.80 1024359.80       0.00  512179.90  512179.90  512179.90       0.00  307.50795         NA            NA     0  16384   8    1   1     1        1         0    0      1 10401873920000  2097152 314998784.0 POSIX      0
Finished            : Thu Nov 14 08:13:44 2019

ior_easy_write

IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Nov 14 07:32:46 2019
Command line        : /cache/xg17i000/x10007/io500-sc-19/bin/ior -w -t 2048k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy/ior_file_easy -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux c0468.ofp
TestID              : 0
StartTime           : Thu Nov 14 07:32:46 2019
Path                : /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy
FS                  : 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.7%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 16384
clients per node    : 8
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 155000 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
stonewalling pairs accessed min: 7844 max: 9613 -- min data: 15.3 GiB mean data: 17.3 GiB time: 300.5s
WARNING: Expected aggregate file size       = 170424302305280000.
WARNING: Stat() of aggregate file size      = 330300164931584.
WARNING: Using actual aggregate bytes moved = 330300164931584.
WARNING: maybe caused by deadlineForStonewalling
write     939818     10158080000 2048.00    0.562165   334.59     0.017622   335.17     0   
Max Write: 939818.47 MiB/sec (985471.09 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      939818.47  939818.47  939818.47       0.00  469909.24  469909.24  469909.24       0.00  335.16982     300.50     964488.40     0  16384   8    1   1     1        1         0    0      1 10401873920000  2097152 314998784.0 POSIX      0
Finished            : Thu Nov 14 07:38:22 2019

ior_hard_read

ior WARNING: fsync() not supported in selected backend.  Using value of 0.
[3181] 2019-11-14 08:17:55 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1a02380 172.30.140.201@verbs-ib C P0-peer00 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3181] 2019-11-14 08:17:55 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1a02380 172.30.140.201@verbs-ib C P0-peer00 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Nov 14 08:17:57 2019
Command line        : /cache/xg17i000/x10007/io500-sc-19/bin/ior -r -R -s 385000 -a IME -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/IOR_file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/stonewall
Machine             : Linux c0468.ofp
TestID              : 0
StartTime           : Thu Nov 14 08:17:57 2019
Path                : /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard
FS                  : 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.9%
[1675] 2019-11-14 08:18:09 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x2c7b150 172.30.140.236@verbs-ib C P3-peer05 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1675] 2019-11-14 08:18:09 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x2c7b150 172.30.140.236@verbs-ib C P3-peer05 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)

Options: 
api                 : IME
apiVersion          : 1.2.2 (1573)
test filename       : /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 385000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 16384
clients per node    : 8
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 269.68 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 296518942720000.
WARNING: Stat() of aggregate file size      = 193356536741888.
WARNING: Using actual aggregate bytes moved = 193356536741888.
read      316927     45.91      45.91      0.402839   581.29     0.139831   581.83     0   
Max Read:  316927.14 MiB/sec (332322.19 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       316927.14  316927.14  316927.14       0.00 7069481.53 7069481.53 7069481.53       0.00  581.83457         NA            NA     0  16384   8    1   0     1        1         0    0 385000    47008    47008 184399168.0 IME      0
Finished            : Thu Nov 14 08:28:17 2019

ior_hard_write

ior WARNING: fsync() not supported in selected backend.  Using value of 0.
[13058] 2019-11-14 07:47:51 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x21b9fc0 172.30.140.210@verbs-ib C P0-peer09 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13058] 2019-11-14 07:47:51 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x21b9fc0 172.30.140.210@verbs-ib C P0-peer09 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Nov 14 07:48:17 2019
Command line        : /cache/xg17i000/x10007/io500-sc-19/bin/ior -w -s 385000 -a IME -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/IOR_file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux c0468.ofp
TestID              : 0
StartTime           : Thu Nov 14 07:48:17 2019
Path                : /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard
FS                  : 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.8%

Options: 
api                 : IME
apiVersion          : 1.2.2 (1573)
test filename       : /cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 385000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 16384
clients per node    : 8
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 269.68 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
stonewalling pairs accessed min: 138313 max: 251054 -- min data: 6.1 GiB mean data: 8.4 GiB time: 306.8s
WARNING: Expected aggregate file size       = 296518942720000.
WARNING: Stat() of aggregate file size      = 193356536741888.
WARNING: Using actual aggregate bytes moved = 193356536741888.
WARNING: maybe caused by deadlineForStonewalling
write     411839     45.91      45.91      2.78       444.25     0.720019   447.75     0   
Max Write: 411838.95 MiB/sec (431844.44 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      411838.95  411838.95  411838.95       0.00 9186615.85 9186615.85 9186615.85       0.00  447.74581     306.77     457988.94     0  16384   8    1   0     1        1         0    0 385000    47008    47008 184399168.0 IME      0
Finished            : Thu Nov 14 07:55:52 2019

mdtest_easy_delete

[1572] 2019-11-14 08:33:20 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1c10630 172.30.140.138@verbs-ib C P3-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1572] 2019-11-14 08:33:20 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1c10630 172.30.140.138@verbs-ib C P3-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
-- started at 11/14/2019 08:33:38 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-r' '-F' '-P' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_easy' '-n' '900000' '-u' '-L' '-a' 'IME' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_easy-stonewall' '-N' '1'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.9%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 14745600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      38916.887      38916.126      38916.819          0.020
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.150          0.150          0.150          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        590.253        590.242        590.243          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          6.675          6.675          6.675          0.000
-- finished at 11/14/2019 08:43:35 --

mdtest_easy_stat

[4363] 2019-11-14 08:13:55 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1ed5630 172.30.140.238@verbs-ib C P3-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[4363] 2019-11-14 08:13:55 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1ed5630 172.30.140.238@verbs-ib C P3-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[4376] 2019-11-14 08:13:55 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x179f4b0 172.30.140.234@verbs-ib C P3-peer03 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[4376] 2019-11-14 08:13:55 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x179f4b0 172.30.140.234@verbs-ib C P3-peer03 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[4379] 2019-11-14 08:13:55 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1b4a1d0 172.30.141.237@verbs-ib C P3-peer06 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[4379] 2019-11-14 08:13:55 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1b4a1d0 172.30.141.237@verbs-ib C P3-peer06 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1510] 2019-11-14 08:13:58 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x154d190 172.30.141.115@verbs-ib C P1-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1510] 2019-11-14 08:13:58 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x154d190 172.30.141.115@verbs-ib C P1-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1515] 2019-11-14 08:13:59 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1ba6120 172.30.141.105@verbs-ib C P0-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1515] 2019-11-14 08:13:59 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1ba6120 172.30.141.105@verbs-ib C P0-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1586] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x15b0630 172.30.140.238@verbs-ib C P3-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1586] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x15b0630 172.30.140.238@verbs-ib C P3-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1589] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1017d90 172.30.141.247@verbs-ib C P4-peer06 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1589] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1017d90 172.30.141.247@verbs-ib C P4-peer06 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1566] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xab6d90 172.30.140.147@verbs-ib C P4-peer06 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1566] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xab6d90 172.30.140.147@verbs-ib C P4-peer06 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1522] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xfb0190 172.30.141.115@verbs-ib C P1-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1522] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xfb0190 172.30.141.115@verbs-ib C P1-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1526] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1c89030 172.30.140.222@verbs-ib C P2-peer01 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1526] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1c89030 172.30.140.222@verbs-ib C P2-peer01 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3217] 2019-11-14 08:14:05 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xd9f400 172.30.140.202@verbs-ib C P0-peer01 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3217] 2019-11-14 08:14:05 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xd9f400 172.30.140.202@verbs-ib C P0-peer01 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1594] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x163e910 172.30.141.135@verbs-ib C P3-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3223] 2019-11-14 08:14:05 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x958630 172.30.140.238@verbs-ib C P3-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1598] 2019-11-14 08:14:00 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xee7a90 172.30.140.239@verbs-ib C P3-peer08 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3223] 2019-11-14 08:14:05 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x958630 172.30.140.238@verbs-ib C P3-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1594] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x163e910 172.30.141.135@verbs-ib C P3-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3229] 2019-11-14 08:14:05 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x25214d0 172.30.140.245@verbs-ib C P4-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1598] 2019-11-14 08:14:00 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xee7a90 172.30.140.239@verbs-ib C P3-peer08 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3229] 2019-11-14 08:14:05 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x25214d0 172.30.140.245@verbs-ib C P4-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1605] 2019-11-14 08:14:01 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x22fb320 172.30.141.148@verbs-ib C P4-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1605] 2019-11-14 08:14:01 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x22fb320 172.30.141.148@verbs-ib C P4-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1646] 2019-11-14 08:14:02 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1d51d30 172.30.140.214@verbs-ib C P1-peer03 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1646] 2019-11-14 08:14:02 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1d51d30 172.30.140.214@verbs-ib C P1-peer03 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1654] 2019-11-14 08:14:02 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x26c4860 172.30.141.103@verbs-ib C P0-peer02 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1654] 2019-11-14 08:14:02 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x26c4860 172.30.141.103@verbs-ib C P0-peer02 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1663] 2019-11-14 08:14:02 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1d99d30 172.30.140.214@verbs-ib C P1-peer03 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1656] 2019-11-14 08:14:02 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1fd35f0 172.30.140.116@verbs-ib C P1-peer05 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1656] 2019-11-14 08:14:02 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1fd35f0 172.30.140.116@verbs-ib C P1-peer05 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[1663] 2019-11-14 08:14:02 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1d99d30 172.30.140.214@verbs-ib C P1-peer03 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3243] 2019-11-14 08:14:05 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xca5010 172.30.141.111@verbs-ib C P1-peer00 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3243] 2019-11-14 08:14:05 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xca5010 172.30.141.111@verbs-ib C P1-peer00 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3246] 2019-11-14 08:14:05 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x7f4630 172.30.140.138@verbs-ib C P3-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3246] 2019-11-14 08:14:05 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x7f4630 172.30.140.138@verbs-ib C P3-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3249] 2019-11-14 08:14:06 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x969ef0 172.30.141.140@verbs-ib C P3-peer09 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3249] 2019-11-14 08:14:06 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x969ef0 172.30.141.140@verbs-ib C P3-peer09 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3251] 2019-11-14 08:14:06 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x8d8320 172.30.141.248@verbs-ib C P4-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[3251] 2019-11-14 08:14:06 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x8d8320 172.30.141.248@verbs-ib C P4-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
-- started at 11/14/2019 08:14:21 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-T' '-F' '-P' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_easy' '-n' '900000' '-u' '-L' '-a' 'IME' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_easy-stonewall' '-N' '1'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.9%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 14745600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     180013.407     180007.319     180012.483          0.224
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        127.608        127.604        127.604          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 11/14/2019 08:16:29 --

mdtest_easy_write

[13923] 2019-11-14 07:38:33 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xe95320 172.30.141.148@verbs-ib C P4-peer07 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13923] 2019-11-14 07:38:33 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xe95320 172.30.141.148@verbs-ib C P4-peer07 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13896] 2019-11-14 07:38:33 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x2102490 172.30.140.123@verbs-ib C P2-peer02 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13896] 2019-11-14 07:38:33 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x2102490 172.30.140.123@verbs-ib C P2-peer02 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13900] 2019-11-14 07:38:33 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x16bc4b0 172.30.141.134@verbs-ib C P3-peer03 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13900] 2019-11-14 07:38:33 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x16bc4b0 172.30.141.134@verbs-ib C P3-peer03 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13916] 2019-11-14 07:38:33 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1125d50 172.30.140.225@verbs-ib C P2-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13912] 2019-11-14 07:38:33 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1672330 172.30.141.130@verbs-ib C P2-peer09 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13916] 2019-11-14 07:38:33 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1125d50 172.30.140.225@verbs-ib C P2-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13912] 2019-11-14 07:38:33 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1672330 172.30.141.130@verbs-ib C P2-peer09 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13939] 2019-11-14 07:38:34 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1ef5490 172.30.140.223@verbs-ib C P2-peer02 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13939] 2019-11-14 07:38:34 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1ef5490 172.30.140.223@verbs-ib C P2-peer02 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13955] 2019-11-14 07:38:34 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x2002d30 172.30.141.214@verbs-ib C P1-peer03 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13954] 2019-11-14 07:38:34 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x17e65f0 172.30.141.116@verbs-ib C P1-peer05 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13954] 2019-11-14 07:38:34 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x17e65f0 172.30.141.116@verbs-ib C P1-peer05 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13955] 2019-11-14 07:38:34 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x2002d30 172.30.141.214@verbs-ib C P1-peer03 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13959] 2019-11-14 07:38:34 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1aaeef0 172.30.141.140@verbs-ib C P3-peer09 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13959] 2019-11-14 07:38:34 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1aaeef0 172.30.141.140@verbs-ib C P3-peer09 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13997] 2019-11-14 07:38:35 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xba7910 172.30.141.235@verbs-ib C P3-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13997] 2019-11-14 07:38:35 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xba7910 172.30.141.235@verbs-ib C P3-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13950] 2019-11-14 07:38:36 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1e864b0 172.30.140.134@verbs-ib C P3-peer03 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13950] 2019-11-14 07:38:36 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1e864b0 172.30.140.134@verbs-ib C P3-peer03 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13988] 2019-11-14 07:38:36 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x22c4490 172.30.140.223@verbs-ib C P2-peer02 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13988] 2019-11-14 07:38:36 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x22c4490 172.30.140.223@verbs-ib C P2-peer02 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13933] 2019-11-14 07:38:36 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1d33930 172.30.140.246@verbs-ib C P4-peer05 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13933] 2019-11-14 07:38:36 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1d33930 172.30.140.246@verbs-ib C P4-peer05 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
-- started at 11/14/2019 07:38:56 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-C' '-F' '-P' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_easy' '-n' '900000' '-u' '-L' '-a' 'IME' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_easy-stonewall' '-N' '1' '-W' '300'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.7%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 14745600000 files
Continue stonewall hit min: 713 max: 1402 avg: 1001.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      51386.737      51386.535      51386.670          0.017
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      54381.774             NA
   Tree creation             :          1.261          1.261          1.261          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        447.011        447.010        447.010          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.823             NA
   Tree creation             :          0.793          0.793          0.793          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 11/14/2019 07:46:24 --

mdtest_hard_delete

-- started at 11/14/2019 08:47:18 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-r' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard' '-n' '950000' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.7%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 15564800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       4737.018       4737.009       4737.012          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          3.646          3.646          3.646          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       1303.939       1303.936       1303.938          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.274          0.274          0.274          0.000
-- finished at 11/14/2019 09:09:03 --

mdtest_hard_read

-- started at 11/14/2019 08:45:04 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-X' '-E' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard' '-n' '950000' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.7%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 15564800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     126682.376     126672.457     126681.379          0.378
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         48.762         48.758         48.758          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 11/14/2019 08:45:54 --

mdtest_hard_stat

-- started at 11/14/2019 08:29:41 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-T' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard' '-n' '950000' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.9%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 15564800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      40467.582      40464.769      40465.109          0.227
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        152.646        152.635        152.644          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 11/14/2019 08:32:13 --

mdtest_hard_write

[6871] 2019-11-14 07:56:55 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1f1d610 172.30.140.227@verbs-ib C P2-peer06 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[6871] 2019-11-14 07:56:55 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1f1d610 172.30.140.227@verbs-ib C P2-peer06 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13005] 2019-11-14 07:56:59 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1fed5f0 172.30.141.116@verbs-ib C P1-peer05 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13005] 2019-11-14 07:56:59 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1fed5f0 172.30.141.116@verbs-ib C P1-peer05 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13007] 2019-11-14 07:56:59 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0xdd5050 172.30.141.233@verbs-ib C P3-peer02 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[13007] 2019-11-14 07:56:59 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0xdd5050 172.30.141.233@verbs-ib C P3-peer02 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[12998] 2019-11-14 07:56:59 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x18b6910 172.30.140.235@verbs-ib C P3-peer04 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[12998] 2019-11-14 07:56:59 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x18b6910 172.30.140.235@verbs-ib C P3-peer04 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[12996] 2019-11-14 07:56:59 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x1b32c10 172.30.140.143@verbs-ib C P4-peer02 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[12996] 2019-11-14 07:56:59 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x1b32c10 172.30.140.143@verbs-ib C P4-peer02 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[8620] 2019-11-14 07:57:04 [ warn] (   im_net_cci_process_connection_connect:01067) Connection request received rc=-54 (Connection reset by peer) (conn@0x21f9350 172.30.141.141@verbs-ib C P4-peer00 st=3 ref=3 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
[8620] 2019-11-14 07:57:04 [ warn] (       im_rpc_connection_callback_common:03453) rc=-54 (Connection reset by peer) (conn@0x21f9350 172.30.141.141@verbs-ib C P4-peer00 st=1 ref=2 cr(tx:rx)=12:24 num(tx:rx)=0:0 TxC=0 rsp(tx:rx)=0:0 CCF=0 tx(p:o:n)=0:0:0 rx(p:a:cw)=0:0:0)
-- started at 11/14/2019 07:57:23 --

mdtest-3.3.0+dev was launched with 16384 total task(s) on 2048 node(s)
Command line used: /cache/xg17i000/x10007/io500-sc-19/bin/mdtest '-C' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard' '-n' '950000' '-x' '/cache/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1' '-W' '300'
Path: /cache/1/xg17i000/x10007/io500-sc-19/datafiles/io500.2019.11.14-07.32.14
FS: 23597.0 TiB   Used FS: 33.9%   Inodes: 12312.0 Mi   Used Inodes: 14.8%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 8 for each phase.
16384 tasks, 15564800000 files
Continue stonewall hit min: 214 max: 377 avg: 283.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      13090.144      13089.982      13090.033          0.018
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      15357.909             NA
   Tree creation             :        222.274        222.274        222.274          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        471.870        471.864        471.868          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.950             NA
   Tree creation             :          0.004          0.004          0.004          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 11/14/2019 08:05:15 --

result_summary

[RESULT] BW   phase 1            ior_easy_write              917.791 GB/s : time 335.17 seconds
[RESULT] IOPS phase 1         mdtest_easy_write               51.387 kiops : time 447.01 seconds
[RESULT] BW   phase 2            ior_hard_write              402.187 GB/s : time 447.75 seconds
[RESULT] IOPS phase 2         mdtest_hard_write               13.090 kiops : time 471.87 seconds
[RESULT] IOPS phase 3                      find              271.760 kiops : time 108.53 seconds
[RESULT] BW   phase 3             ior_easy_read             1000.350 GB/s : time 307.51 seconds
[RESULT] IOPS phase 4          mdtest_easy_stat              180.013 kiops : time 127.61 seconds
[RESULT] BW   phase 4             ior_hard_read              309.499 GB/s : time 581.83 seconds
[RESULT] IOPS phase 5          mdtest_hard_stat               40.468 kiops : time 152.65 seconds
[RESULT] IOPS phase 6        mdtest_easy_delete               38.917 kiops : time 590.25 seconds
[RESULT] IOPS phase 7          mdtest_hard_read              126.682 kiops : time  48.76 seconds
[RESULT] IOPS phase 8        mdtest_hard_delete                4.737 kiops : time 1382.85 seconds
[SCORE] Bandwidth 581.428 GB/s : IOPS 48.5952 kiops : TOTAL 168.091