Oakforest-PACS

Institution JCAHPC
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version
Client Kernel Version

DATA SERVER

Storage Type NVMe-SSD
Volatile Memory
Storage Interface NVMe
Network Omnipath
Software Version
OS Version

INFORMATION

Client Nodes 512
Client Total Procs 4,096
Metadata Nodes 1
Metadata Storage Devices 12
Data Nodes 25
Data Storage Devices 1,200

METADATA

Easy Write 12.18 kIOP/s
Easy Stat 197.20 kIOP/s
Easy Delete 84.11 kIOP/s
Hard Write 11.88 kIOP/s
Hard Read 123.41 kIOP/s
Hard Stat 128.04 kIOP/s
Hard Delete 23.11 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jun 25 12:47:49 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a IME
Machine             : Linux c0229.ofp
TestID              : 0
StartTime           : Thu Jun 25 12:47:49 2020
Path                : /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy
FS                  : 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Options: 
api                 : IME
apiVersion          : 1.3.2.1 (132037)
test filename       : /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 512
tasks               : 4096
clients per node    : 8
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 38750 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 42606075576320000.
WARNING: Stat() of aggregate file size      = 176789443837952.
WARNING: Using actual aggregate bytes moved = 176789443837952.
read      587934     294055     0.000058    10158080000 2048.00    0.069487   286.68     0.039780   286.77     0   
Max Read:  587934.39 MiB/sec (616493.89 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       587934.39  587934.39  587934.39       0.00  293967.20  293967.20  293967.20       0.00  286.76593         NA            NA     0   4096   8    1   1     1        1         0    0      1 10401873920000  2097152 168599552.0 IME      0
Finished            : Thu Jun 25 12:52:36 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jun 25 12:18:39 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a IME
Machine             : Linux c0229.ofp
TestID              : 0
StartTime           : Thu Jun 25 12:18:39 2020
Path                : /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy
FS                  : 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.0%

Options: 
api                 : IME
apiVersion          : 1.3.2.1 (132037)
test filename       : /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 512
tasks               : 4096
clients per node    : 8
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 38750 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 5286 max: 20581 -- min data: 10.3 GiB mean data: 34.6 GiB time: 300.2s
WARNING: Expected aggregate file size       = 42606075576320000.
WARNING: Stat() of aggregate file size      = 176789443837952.
WARNING: Using actual aggregate bytes moved = 176789443837952.
WARNING: maybe caused by deadlineForStonewalling
write     341965     172380     0.000068    10158080000 2048.00    0.379008   489.03     3.63       493.03     0   
Max Write: 341965.05 MiB/sec (358576.35 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      341965.05  341965.05  341965.05       0.00  170982.53  170982.53  170982.53       0.00  493.03152     300.17     483423.69     0   4096   8    1   1     1        1         0    0      1 10401873920000  2097152 168599552.0 IME      0
Finished            : Thu Jun 25 12:26:52 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jun 25 12:53:01 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a IME
Machine             : Linux c0229.ofp
TestID              : 0
StartTime           : Thu Jun 25 12:53:01 2020
Path                : /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard
FS                  : 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Options: 
api                 : IME
apiVersion          : 1.3.2.1 (132037)
test filename       : /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 512
tasks               : 4096
clients per node    : 8
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 1751.18 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 752012 max: 752012 -- min data: 32.9 GiB mean data: 32.9 GiB time: 535.6s
WARNING: Expected aggregate file size       = 1925447680000000.
WARNING: Stat() of aggregate file size      = 144795976073216.
WARNING: Using actual aggregate bytes moved = 144795976073216.
read      257000     5736692    536.91      45.91      45.91      0.132194   536.94     0.278699   537.31     0   
Max Read:  257000.39 MiB/sec (269484.44 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       257000.39  257000.39  257000.39       0.00 5732735.64 5732735.64 5732735.64       0.00  537.30738     535.57     257833.11     0   4096   8    1   0     1        1         0    0 10000000    47008    47008 138088208.0 IME      0
Finished            : Thu Jun 25 13:01:58 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jun 25 12:33:19 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a IME
Machine             : Linux c0229.ofp
TestID              : 0
StartTime           : Thu Jun 25 12:33:19 2020
Path                : /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard
FS                  : 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Options: 
api                 : IME
apiVersion          : 1.3.2.1 (132037)
test filename       : /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 512
tasks               : 4096
clients per node    : 8
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 1751.18 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 276175 max: 752012 -- min data: 12.1 GiB mean data: 26.6 GiB time: 300.1s
WARNING: Expected aggregate file size       = 1925447680000000.
WARNING: Stat() of aggregate file size      = 144795976073216.
WARNING: Using actual aggregate bytes moved = 144795976073216.
WARNING: maybe caused by deadlineForStonewalling
write     288716     6447430    341.06      45.91      45.91      0.086754   477.75     0.442341   478.28     0   
Max Write: 288715.92 MiB/sec (302740.59 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      288715.92  288715.92  288715.92       0.00 6440192.87 6440192.87 6440192.87       0.00  478.28399     300.07     371253.80     0   4096   8    1   0     1        1         0    0 10000000    47008    47008 138088208.0 IME      0
Finished            : Thu Jun 25 12:41:18 2020
mdtest-easy-delete
-- started at 06/25/2020 13:02:32 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy-stonewall' '-r' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      84110.541      84101.089      84109.620          0.262
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.472          0.472          0.472          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :         55.960         55.954         55.954          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.117          2.117          2.117          0.000
-- finished at 06/25/2020 13:03:31 --

mdtest-easy-stat
-- started at 06/25/2020 12:52:37 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy-stonewall' '-T' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     197207.423     197160.160     197205.284          2.102
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         23.870         23.865         23.865          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/25/2020 12:53:00 --

mdtest-easy-write
-- started at 06/25/2020 12:26:53 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.0%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files
Continue stonewall hit min: 653 max: 1149 avg: 893.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      12181.948      12181.734      12181.874          0.024
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      12184.799             NA
   Tree creation             :         17.077         17.077         17.077          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        386.341        386.334        386.337          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.259             NA
   Tree creation             :          0.059          0.059          0.059          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/25/2020 12:33:19 --

mdtest-hard-delete
-- started at 06/25/2020 13:04:06 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall' '-r' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      23114.925      23114.002      23114.427          0.063
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          7.963          7.963          7.963          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        183.943        183.935        183.939          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.126          0.126          0.126          0.000
-- finished at 06/25/2020 13:07:10 --

mdtest-hard-read
-- started at 06/25/2020 13:03:31 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     123432.148     123387.291     123407.846          2.301
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         34.458         34.445         34.452          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/25/2020 13:04:05 --

mdtest-hard-stat
-- started at 06/25/2020 13:01:59 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall' '-T' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     128045.330     128029.395     128038.438          0.826
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         33.208         33.204         33.206          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/25/2020 13:02:32 --

mdtest-hard-write
-- started at 06/25/2020 12:41:18 --

mdtest-3.3.0+dev was launched with 4096 total task(s) on 512 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard' '-x' '/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'IME'
Path: /cache/1/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app
FS: 23597.0 TiB   Used FS: 42.9%   Inodes: 12312.0 Mi   Used Inodes: 18.1%

Nodemap: 1111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 8 for each phase.
4096 tasks, 4096000000 files
Continue stonewall hit min: 702 max: 1038 avg: 874.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      11878.824      11878.679      11878.751          0.022
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      11920.603             NA
   Tree creation             :        821.265        821.265        821.265          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        357.923        357.918        357.920          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.404             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/25/2020 12:47:16 --

result
version         = io500-isc20
config-hash     = B5BAD2D2
result-dir      = /work/xg17i000/x10007/io500-app/results/2020.06.25-12.17.18-app
; START 2020-06-25 12:18:39


[ior-easy-write]
t_start         = 2020-06-25 12:18:39
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a IME
throughput-stonewall = 1177276.82
score           = 333.950249
t_delta         = 493.5454
t_end           = 2020-06-25 12:26:52

[mdtest-easy-write]
t_start         = 2020-06-25 12:26:52
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy-stonewall -C -Y -W 300 -a IME
rate-stonewall  = 12.184799
score           = 12.181884
t_delta         = 386.7948
t_end           = 2020-06-25 12:33:19

[timestamp]
t_start         = 2020-06-25 12:33:19
t_delta         = 0.0098
t_end           = 2020-06-25 12:33:19

[ior-hard-write]
t_start         = 2020-06-25 12:33:19
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a IME
throughput-stonewall = 931666.93
score           = 281.949142
t_delta         = 478.8196
t_end           = 2020-06-25 12:41:18

[mdtest-hard-write]
t_start         = 2020-06-25 12:41:18
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall -C -Y -W 300 -a IME
rate-stonewall  = 11.920603
score           = 11.878759
t_delta         = 358.3157
t_end           = 2020-06-25 12:47:16

[find]
t_start         = 2020-06-25 12:47:16
exe             = ./pfind /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app -newer /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/timestampfile -size 3901c -name *01* -C -q 10000
nproc           = 1024
found           = 163360
total-files     = 8962054
score           = 273.757568
t_delta         = 32.8141
t_end           = 2020-06-25 12:47:49

[ior-easy-read]
t_start         = 2020-06-25 12:47:49
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a IME
score           = 574.154678
t_delta         = 287.1683
t_end           = 2020-06-25 12:52:36

[mdtest-easy-stat]
t_start         = 2020-06-25 12:52:36
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy-stonewall -T -a IME
score           = 197.204876
t_delta         = 24.2418
t_end           = 2020-06-25 12:53:01

[ior-hard-read]
t_start         = 2020-06-25 12:53:01
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/file -O stoneWallingStatusFile=/cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a IME
score           = 250.976940
t_delta         = 537.8295
t_end           = 2020-06-25 13:01:58

[mdtest-hard-stat]
t_start         = 2020-06-25 13:01:58
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall -T -a IME
score           = 128.038640
t_delta         = 33.6178
t_end           = 2020-06-25 13:02:32

[mdtest-easy-delete]
t_start         = 2020-06-25 13:02:32
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-easy-stonewall -r -a IME
score           = 84.109743
t_delta         = 58.5716
t_end           = 2020-06-25 13:03:31

[mdtest-hard-read]
t_start         = 2020-06-25 13:03:31
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall -E -X -a IME
score           = 123.407475
t_delta         = 34.9685
t_end           = 2020-06-25 13:04:05

[mdtest-hard-delete]
t_start         = 2020-06-25 13:04:05
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard -x /cache/xg17i000/x10007/io500-app/datafiles/2020.06.25-12.17.18-app/mdtest-hard-stonewall -r -a IME
score           = 23.114441
t_delta         = 184.5878
t_end           = 2020-06-25 13:07:10

[SCORE]
MD              = 62.736847
BW              = 341.294130
SCORE           = 146.327433 
hash            = AFC293E5
; END 2020-06-25 13:07:11
result_summary
IO500 version io500-isc20
[RESULT]       ior-easy-write      333.950249 GiB/s  : time 493.545 seconds
[RESULT]    mdtest-easy-write       12.181884 kIOPS : time 386.795 seconds
[RESULT]       ior-hard-write      281.949142 GiB/s  : time 478.820 seconds
[RESULT]    mdtest-hard-write       11.878759 kIOPS : time 358.316 seconds
[RESULT]                 find      273.757568 kIOPS : time 32.814 seconds
[RESULT]        ior-easy-read      574.154678 GiB/s  : time 287.168 seconds
[RESULT]     mdtest-easy-stat      197.204876 kIOPS : time 24.242 seconds
[RESULT]        ior-hard-read      250.976940 GiB/s  : time 537.829 seconds
[RESULT]     mdtest-hard-stat      128.038640 kIOPS : time 33.618 seconds
[RESULT]   mdtest-easy-delete       84.109743 kIOPS : time 58.572 seconds
[RESULT]     mdtest-hard-read      123.407475 kIOPS : time 34.968 seconds
[RESULT]   mdtest-hard-delete       23.114441 kIOPS : time 184.588 seconds
[SCORE] Bandwidth 341.294130 GB/s : IOPS 62.736847 kiops : TOTAL 146.327433