Cedar

Institution Simon Fraser University
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.12.1.el7.x86_64

DATA SERVER

Storage Type NL-SAS7.2K
Volatile Memory 128GB
Storage Interface SAS
Network Omni-Path
Software Version 2.12.4
OS Version CentOS 7.7

INFORMATION

Client Nodes 288
Client Total Procs 288
Metadata Nodes 2
Metadata Storage Devices 1
Data Nodes 12
Data Storage Devices 48

METADATA

Easy Write 13.80 kIOP/s
Easy Stat 90.03 kIOP/s
Easy Delete 31.76 kIOP/s
Hard Write 14.39 kIOP/s
Hard Read 30.37 kIOP/s
Hard Stat 43.65 kIOP/s
Hard Delete 29.41 kIOP/s

Submitted Files

ior_easy_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Mar 22 22:00:25 2020
Command line        : /home/liu/project/io500/io-500-dev-master/bin/ior -r -R -t 2048k -b 65536m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy/ior_file_easy -O stoneWallingStatusFile=/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy/stonewall
Machine             : Linux cdr2.int.cedar.computecanada.ca
TestID              : 0
StartTime           : Sun Mar 22 22:00:25 2020
Path                : /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy
FS                  : 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 37.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 288
tasks               : 288
clients per node    : 1
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 64 GiB
aggregate filesize  : 18 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
read      41142      20573      0.013999    67108864   2048.00    0.010456   458.72     0.034705   458.76     0   
Max Read:  41141.99 MiB/sec (43140.51 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        41141.99   41141.99   41141.99       0.00   20571.00   20571.00   20571.00       0.00  458.76163         NA            NA     0    288   1    1   1     1        1         0    0      1 68719476736  2097152 18874368.0 POSIX      0
Finished            : Sun Mar 22 22:08:04 2020
ior_easy_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Mar 22 21:26:08 2020
Command line        : /home/liu/project/io500/io-500-dev-master/bin/ior -w -t 2048k -b 65536m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy/ior_file_easy -O stoneWallingStatusFile=/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux cdr2.int.cedar.computecanada.ca
TestID              : 0
StartTime           : Sun Mar 22 21:26:08 2020
Path                : /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy
FS                  : 12510.2 TiB   Used FS: 71.4%   Inodes: 1760.0 Mi   Used Inodes: 36.3%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 288
tasks               : 288
clients per node    : 1
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 64 GiB
aggregate filesize  : 18 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 6152 max: 32768 -- min data: 12.0 GiB mean data: 40.0 GiB time: 301.5s
write     32578      16290      0.009201    67108864   2048.00    0.007106   579.33     0.026925   579.37     0   
Max Write: 32577.51 MiB/sec (34159.99 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       32577.51   32577.51   32577.51       0.00   16288.75   16288.75   16288.75       0.00  579.36805     301.46      39154.54     0    288   1    1   1     1        1         0    0      1 68719476736  2097152 18874368.0 POSIX      0
Finished            : Sun Mar 22 21:35:48 2020
ior_hard_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Mar 22 22:09:49 2020
Command line        : /home/liu/project/io500/io-500-dev-master/bin/ior -r -R -s 18432 -a POSIX -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard/IOR_file -O stoneWallingStatusFile=/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard/stonewall
Machine             : Linux cdr2.int.cedar.computecanada.ca
TestID              : 0
StartTime           : Sun Mar 22 22:09:49 2020
Path                : /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard
FS                  : 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 37.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 18432
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 288
tasks               : 288
clients per node    : 1
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 232.40 GiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
read      5379       120131     44.18       45.91      45.91      0.042956   44.19      0.014756   44.25      0   
Max Read:  5378.54 MiB/sec (5639.80 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         5378.54    5378.54    5378.54       0.00  119975.42  119975.42  119975.42       0.00   44.24586         NA            NA     0    288   1    1   0     1        1         0    0  18432    47008    47008  237978.0 POSIX      0
Finished            : Sun Mar 22 22:10:33 2020
ior_hard_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Mar 22 21:45:51 2020
Command line        : /home/liu/project/io500/io-500-dev-master/bin/ior -w -s 18432 -a POSIX -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard/IOR_file -O stoneWallingStatusFile=/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux cdr2.int.cedar.computecanada.ca
TestID              : 0
StartTime           : Sun Mar 22 21:45:51 2020
Path                : /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard
FS                  : 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 36.8%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 18432
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 288
tasks               : 288
clients per node    : 1
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 232.40 GiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 8269 max: 18432 -- min data: 0.4 GiB mean data: 0.6 GiB time: 300.1s
write     524.93     12762      346.65      45.91      45.91      37.37      415.96     0.027000   453.35     0   
Max Write: 524.93 MiB/sec (550.43 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write         524.93     524.93     524.93       0.00   11709.23   11709.23   11709.23       0.00  453.35315     300.06        615.66     0    288   1    1   0     1        1         0    0  18432    47008    47008  237978.0 POSIX      0
Finished            : Sun Mar 22 21:53:24 2020
mdtest_easy_delete
-- started at 03/22/2020 22:12:57 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-r' '-F' '-P' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_easy' '-n' '64000' '-u' '-L' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_easy-stonewall' '-N' '1'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 37.1%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      31763.655      31761.788      31762.575          0.555
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          3.214          3.214          3.214          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        254.153        254.138        254.147          0.004
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.311          0.311          0.311          0.000
-- finished at 03/22/2020 22:17:11 --

mdtest_easy_stat
-- started at 03/22/2020 22:08:12 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-T' '-F' '-P' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_easy' '-n' '64000' '-u' '-L' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_easy-stonewall' '-N' '1'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 37.1%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      90027.413      90018.501      90022.677          2.478
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         89.674         89.665         89.670          0.002
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 03/22/2020 22:09:42 --

mdtest_easy_write
-- started at 03/22/2020 21:35:55 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-C' '-F' '-P' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_easy' '-n' '64000' '-u' '-L' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_easy-stonewall' '-N' '1' '-W' '300'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 36.3%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files
Continue stonewall hit min: 5392 max: 28029 avg: 13534.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      13804.026      13803.674      13803.862          0.113
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      12254.718             NA
   Tree creation             :         95.314         95.314         95.314          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        584.797        584.782        584.789          0.005
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        318.075             NA
   Tree creation             :          0.010          0.010          0.010          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 03/22/2020 21:45:40 --

mdtest_hard_delete
-- started at 03/22/2020 22:20:30 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-r' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard' '-n' '64000' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 36.6%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      29415.483      29413.502      29414.657          0.465
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          5.326          5.326          5.326          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        191.775        191.762        191.767          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.188          0.188          0.188          0.000
-- finished at 03/22/2020 22:23:42 --

mdtest_hard_read
-- started at 03/22/2020 22:17:18 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-X' '-E' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard' '-n' '64000' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 36.6%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      30370.177      30369.080      30369.585          0.236
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        185.740        185.734        185.737          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 03/22/2020 22:20:24 --

mdtest_hard_stat
-- started at 03/22/2020 22:10:39 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-T' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard' '-n' '64000' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 37.1%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      43651.133      43647.599      43648.699          0.883
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        129.234        129.224        129.231          0.003
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 03/22/2020 22:12:49 --

mdtest_hard_write
-- started at 03/22/2020 21:53:31 --

mdtest-3.3.0+dev was launched with 288 total task(s) on 288 node(s)
Command line used: /home/liu/project/io500/io-500-dev-master/bin/mdtest '-C' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard' '-n' '64000' '-x' '/home/liu/project/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1' '-W' '300'
Path: /project/6001493/io500/io-500-dev-master/datafiles/io500.2020.03.22-21.26.02
FS: 12510.2 TiB   Used FS: 71.5%   Inodes: 1760.0 Mi   Used Inodes: 36.8%

Nodemap: 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2137 Shifting ranks by 1 for each phase.
288 tasks, 18432000 files
Continue stonewall hit min: 9403 max: 19586 avg: 13343.7 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      14388.551      14388.318      14388.475          0.087
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      11829.540             NA
   Tree creation             :        819.243        819.243        819.243          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        392.038        392.032        392.034          0.002
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        324.863             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 03/22/2020 22:00:03 --

result_summary
[RESULT] BW   phase 1            ior_easy_write               31.815 GB/s : time 579.33 seconds
[RESULT] IOPS phase 1         mdtest_easy_write               13.804 kiops : time 584.80 seconds
[RESULT] BW   phase 2            ior_hard_write                0.513 GB/s : time 415.96 seconds
[RESULT] IOPS phase 2         mdtest_hard_write               14.389 kiops : time 392.04 seconds
[RESULT] IOPS phase 3                      find              866.520 kiops : time  15.83 seconds
[RESULT] BW   phase 3             ior_easy_read               40.178 GB/s : time 458.72 seconds
[RESULT] IOPS phase 4          mdtest_easy_stat               90.027 kiops : time  89.67 seconds
[RESULT] BW   phase 4             ior_hard_read                5.253 GB/s : time  44.19 seconds
[RESULT] IOPS phase 5          mdtest_hard_stat               43.651 kiops : time 129.23 seconds
[RESULT] IOPS phase 6        mdtest_easy_delete               31.764 kiops : time 254.15 seconds
[RESULT] IOPS phase 7          mdtest_hard_read               30.370 kiops : time 185.74 seconds
[RESULT] IOPS phase 8        mdtest_hard_delete               29.415 kiops : time 198.75 seconds
[SCORE] Bandwidth 7.65955 GB/s : IOPS 45.75 kiops : TOTAL 18.7196