SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.7.0
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 600
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 20
Data Storage Devices 80

METADATA

Easy Write 29.36 kIOP/s
Easy Stat 1,073.90 kIOP/s
Easy Delete 29.22 kIOP/s
Hard Write 10.95 kIOP/s
Hard Read 95.81 kIOP/s
Hard Stat 143.15 kIOP/s
Hard Delete 17.79 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 22:05:12 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 22:05:12 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-easy
FS                  : 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 5.72 TiB
aggregate filesize  : 3433.23 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 3774873600000000.
WARNING: Stat() of aggregate file size      = 6321654988800.
WARNING: Using actual aggregate bytes moved = 6321654988800.
read      22902      715.71     0.001404    6144000000 32768      0.008918   263.23     0.002804   263.25     0   
Max Read:  22901.79 MiB/sec (24014.27 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        22901.79   22901.79   22901.79       0.00     715.68     715.68     715.68       0.00  263.24578         NA            NA     0    600  60    1   1     1        1         0    0      1 6291456000000 33554432 6028800.0 POSIX      0
Finished            : Tue Jun  2 22:09:36 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 21:20:42 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 21:20:42 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-easy
FS                  : 50.0 TiB   Used FS: 64.3%   Inodes: 858.3 Mi   Used Inodes: 3.7%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 5.72 TiB
aggregate filesize  : 3433.23 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 193 max: 314 -- min data: 6.0 GiB mean data: 7.7 GiB time: 300.8s
WARNING: Expected aggregate file size       = 3774873600000000.
WARNING: Stat() of aggregate file size      = 6321654988800.
WARNING: Using actual aggregate bytes moved = 6321654988800.
WARNING: maybe caused by deadlineForStonewalling
write     16039      501.33     0.001923    6144000000 32768      0.073575   375.80     0.019483   375.89     0   
Max Write: 16038.54 MiB/sec (16817.63 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       16038.54   16038.54   16038.54       0.00     501.20     501.20     501.20       0.00  375.89455     300.80      15814.83     0    600  60    1   1     1        1         0    0      1 6291456000000 33554432 6028800.0 POSIX      0
Finished            : Tue Jun  2 21:26:58 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 22:09:49 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -r -R -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 22:09:49 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-hard
FS                  : 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1800000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 46.17 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 180821 max: 180821 -- min data: 7.9 GiB mean data: 7.9 GiB time: 495.2s
WARNING: Expected aggregate file size       = 50768640000000.
WARNING: Stat() of aggregate file size      = 5100020140800.
WARNING: Using actual aggregate bytes moved = 5100020140800.
read      9820       219053     495.28      45.91      45.91      0.000605   495.28     0.000266   495.28     0   
Max Read:  9820.19 MiB/sec (10297.21 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         9820.19    9820.19    9820.19       0.00  219052.33  219052.33  219052.33       0.00  495.28165     495.21       9821.59     0    600  60    1   0     1        1         0    0 1800000    47008    47008 4863758.0 POSIX      0
Finished            : Tue Jun  2 22:18:05 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 21:41:27 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -w -D 300 -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 21:41:27 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-hard
FS                  : 50.0 TiB   Used FS: 75.9%   Inodes: 858.3 Mi   Used Inodes: 6.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1800000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 46.17 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 21 max: 180821 -- min data: 0.0 GiB mean data: 0.1 GiB time: 300.1s
WARNING: Expected aggregate file size       = 50768640000000.
WARNING: Stat() of aggregate file size      = 5100020140800.
WARNING: Using actual aggregate bytes moved = 5100020140800.
WARNING: maybe caused by deadlineForStonewalling
write     9681       215954     438.86      45.91      45.91      0.002525   502.39     0.000223   502.39     0   
Max Write: 9681.24 MiB/sec (10151.52 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        9681.24    9681.24    9681.24       0.00  215952.94  215952.94  215952.94       0.00  502.39000     300.05        280.95     0    600  60    1   0     1        1         0    0 1800000    47008    47008 4863758.0 POSIX      0
Finished            : Tue Jun  2 21:49:50 2020
mdtest-easy-delete
-- started at 06/02/2020 22:18:38 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '24000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-easy' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 14400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      29221.734      29221.729      29221.732          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.694          2.694          2.694          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        492.784        492.784        492.784          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.371          0.371          0.371          0.000
-- finished at 06/02/2020 22:26:52 --

mdtest-easy-stat
-- started at 06/02/2020 22:09:36 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '24000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-easy' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 14400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    1073896.694    1073892.532    1073895.986          0.403
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         13.409         13.409         13.409          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:09:49 --

mdtest-easy-write
-- started at 06/02/2020 21:26:58 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '24000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-easy' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 75.8%   Inodes: 858.3 Mi   Used Inodes: 4.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 14400000 files
Continue stonewall hit min: 233 max: 24000 avg: 15984.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      29360.606      29360.588      29360.594          0.004
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      31867.536             NA
   Tree creation             :         16.918         16.918         16.918          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        490.453        490.453        490.453          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.946             NA
   Tree creation             :          0.059          0.059          0.059          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 21:41:27 --

mdtest-hard-delete
-- started at 06/02/2020 22:27:42 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.2%   Inodes: 858.3 Mi   Used Inodes: 5.5%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      17791.361      17791.358      17791.360          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.655          0.655          0.655          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        269.726        269.726        269.726          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.526          1.526          1.526          0.000
-- finished at 06/02/2020 22:32:14 --

mdtest-hard-read
-- started at 06/02/2020 22:26:52 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.2%   Inodes: 858.3 Mi   Used Inodes: 5.5%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      95809.502      95797.847      95809.459          0.475
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         50.093         50.087         50.087          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:27:42 --

mdtest-hard-stat
-- started at 06/02/2020 22:18:05 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     143148.156     143148.020     143148.098          0.024
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         33.523         33.523         33.523          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:18:38 --

mdtest-hard-write
-- started at 06/02/2020 21:49:50 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.2%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files
Continue stonewall hit min: 7794 max: 7998 avg: 7930.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      10945.819      10945.816      10945.817          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      15848.592             NA
   Tree creation             :        808.463        808.463        808.463          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        438.414        438.414        438.414          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.231             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:04:00 --

result
version         = io500-isc20
config-hash     = 8798F7A
result-dir      = ./results/2020.06.02-19.41.54-app
; START 2020-06-02 21:20:42


[ior-easy-write]
t_start         = 2020-06-02 21:20:42
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 7415.31
score           = 15.662637
t_delta         = 376.0656
t_end           = 2020-06-02 21:26:58

[mdtest-easy-write]
t_start         = 2020-06-02 21:26:58
exe             = ./mdtest -n 24000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-19.41.54-app/mdtest-easy -x ./datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 31.867536
score           = 29.360593
t_delta         = 869.6763
t_end           = 2020-06-02 21:41:27

[timestamp]
t_start         = 2020-06-02 21:41:27
t_delta         = 0.0010
t_end           = 2020-06-02 21:41:27

[ior-hard-write]
t_start         = 2020-06-02 21:41:27
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -w -D 300 -a POSIX --posix.odirect
throughput-stonewall = 98.32
score           = 9.454336
t_delta         = 502.4056
t_end           = 2020-06-02 21:49:50

[mdtest-hard-write]
t_start         = 2020-06-02 21:49:50
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 15.848592
score           = 10.945818
t_delta         = 850.5657
t_end           = 2020-06-02 22:04:00

[find]
t_start         = 2020-06-02 22:04:00
exe             = ./pfind ./datadir/2020.06.02-19.41.54-app -newer ./datadir/2020.06.02-19.41.54-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 128645
total-files     = 19199406
score           = 268.944060
t_delta         = 71.4426
t_end           = 2020-06-02 22:05:12

[ior-easy-read]
t_start         = 2020-06-02 22:05:12
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -r -R -a POSIX
score           = 22.365031
t_delta         = 264.0148
t_end           = 2020-06-02 22:09:36

[mdtest-easy-stat]
t_start         = 2020-06-02 22:09:36
exe             = ./mdtest -n 24000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-19.41.54-app/mdtest-easy -x ./datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall -T -a POSIX
score           = 1073.896694
t_delta         = 13.4482
t_end           = 2020-06-02 22:09:49

[ior-hard-read]
t_start         = 2020-06-02 22:09:49
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -r -R -a POSIX --posix.odirect
score           = 9.590026
t_delta         = 495.3104
t_end           = 2020-06-02 22:18:05

[mdtest-hard-stat]
t_start         = 2020-06-02 22:18:05
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -T -a POSIX
score           = 143.148130
t_delta         = 33.5448
t_end           = 2020-06-02 22:18:38

[mdtest-easy-delete]
t_start         = 2020-06-02 22:18:38
exe             = ./mdtest -n 24000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-19.41.54-app/mdtest-easy -x ./datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall -r -a POSIX
score           = 29.221732
t_delta         = 494.0693
t_end           = 2020-06-02 22:26:52

[mdtest-hard-read]
t_start         = 2020-06-02 22:26:52
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 95.809502
t_delta         = 50.1099
t_end           = 2020-06-02 22:27:42

[mdtest-hard-delete]
t_start         = 2020-06-02 22:27:42
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -r -a POSIX
score           = 17.791361
t_delta         = 271.3160
t_end           = 2020-06-02 22:32:14

[SCORE]
MD              = 71.218588
BW              = 13.349694
SCORE           = 30.834175 
hash            = 17F1E90D
; END 2020-06-02 22:32:18
result_summary
IO500 version io500-isc20
[RESULT]       ior-easy-write       15.662637 GiB/s  : time 376.066 seconds
[RESULT]    mdtest-easy-write       29.360593 kIOPS : time 869.676 seconds
[RESULT]       ior-hard-write        9.454336 GiB/s  : time 502.406 seconds
[RESULT]    mdtest-hard-write       10.945818 kIOPS : time 850.566 seconds
[RESULT]                 find      268.944060 kIOPS : time 71.443 seconds
[RESULT]        ior-easy-read       22.365031 GiB/s  : time 264.015 seconds
[RESULT]     mdtest-easy-stat     1073.896694 kIOPS : time 13.448 seconds
[RESULT]        ior-hard-read        9.590026 GiB/s  : time 495.310 seconds
[RESULT]     mdtest-hard-stat      143.148130 kIOPS : time 33.545 seconds
[RESULT]   mdtest-easy-delete       29.221732 kIOPS : time 494.069 seconds
[RESULT]     mdtest-hard-read       95.809502 kIOPS : time 50.110 seconds
[RESULT]   mdtest-hard-delete       17.791361 kIOPS : time 271.316 seconds
[SCORE] Bandwidth 13.349694 GB/s : IOPS 71.218588 kiops : TOTAL 30.834175