SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.7.0
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 600

METADATA

Easy Write 29.36 kIOP/s
Easy Stat 1,073.90 kIOP/s
Easy Delete 29.22 kIOP/s
Hard Write 10.95 kIOP/s
Hard Read 95.81 kIOP/s
Hard Stat 143.15 kIOP/s
Hard Delete 17.79 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 22:05:12 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 22:05:12 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-easy
FS                  : 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 5.72 TiB
aggregate filesize  : 3433.23 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 3774873600000000.
WARNING: Stat() of aggregate file size      = 6321654988800.
WARNING: Using actual aggregate bytes moved = 6321654988800.
read      22902      715.71     0.001404    6144000000 32768      0.008918   263.23     0.002804   263.25     0   
Max Read:  22901.79 MiB/sec (24014.27 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        22901.79   22901.79   22901.79       0.00     715.68     715.68     715.68       0.00  263.24578         NA            NA     0    600  60    1   1     1        1         0    0      1 6291456000000 33554432 6028800.0 POSIX      0
Finished            : Tue Jun  2 22:09:36 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 21:20:42 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 21:20:42 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-easy
FS                  : 50.0 TiB   Used FS: 64.3%   Inodes: 858.3 Mi   Used Inodes: 3.7%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 5.72 TiB
aggregate filesize  : 3433.23 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 193 max: 314 -- min data: 6.0 GiB mean data: 7.7 GiB time: 300.8s
WARNING: Expected aggregate file size       = 3774873600000000.
WARNING: Stat() of aggregate file size      = 6321654988800.
WARNING: Using actual aggregate bytes moved = 6321654988800.
WARNING: maybe caused by deadlineForStonewalling
write     16039      501.33     0.001923    6144000000 32768      0.073575   375.80     0.019483   375.89     0   
Max Write: 16038.54 MiB/sec (16817.63 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       16038.54   16038.54   16038.54       0.00     501.20     501.20     501.20       0.00  375.89455     300.80      15814.83     0    600  60    1   1     1        1         0    0      1 6291456000000 33554432 6028800.0 POSIX      0
Finished            : Tue Jun  2 21:26:58 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 22:09:49 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -r -R -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 22:09:49 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-hard
FS                  : 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1800000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 46.17 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 180821 max: 180821 -- min data: 7.9 GiB mean data: 7.9 GiB time: 495.2s
WARNING: Expected aggregate file size       = 50768640000000.
WARNING: Stat() of aggregate file size      = 5100020140800.
WARNING: Using actual aggregate bytes moved = 5100020140800.
read      9820       219053     495.28      45.91      45.91      0.000605   495.28     0.000266   495.28     0   
Max Read:  9820.19 MiB/sec (10297.21 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         9820.19    9820.19    9820.19       0.00  219052.33  219052.33  219052.33       0.00  495.28165     495.21       9821.59     0    600  60    1   0     1        1         0    0 1800000    47008    47008 4863758.0 POSIX      0
Finished            : Tue Jun  2 22:18:05 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 21:41:27 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -w -D 300 -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 21:41:27 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app/ior-hard
FS                  : 50.0 TiB   Used FS: 75.9%   Inodes: 858.3 Mi   Used Inodes: 6.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-19.41.54-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1800000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 46.17 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 21 max: 180821 -- min data: 0.0 GiB mean data: 0.1 GiB time: 300.1s
WARNING: Expected aggregate file size       = 50768640000000.
WARNING: Stat() of aggregate file size      = 5100020140800.
WARNING: Using actual aggregate bytes moved = 5100020140800.
WARNING: maybe caused by deadlineForStonewalling
write     9681       215954     438.86      45.91      45.91      0.002525   502.39     0.000223   502.39     0   
Max Write: 9681.24 MiB/sec (10151.52 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        9681.24    9681.24    9681.24       0.00  215952.94  215952.94  215952.94       0.00  502.39000     300.05        280.95     0    600  60    1   0     1        1         0    0 1800000    47008    47008 4863758.0 POSIX      0
Finished            : Tue Jun  2 21:49:50 2020
mdtest-easy-delete
-- started at 06/02/2020 22:18:38 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '24000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-easy' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 14400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      29221.734      29221.729      29221.732          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.694          2.694          2.694          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        492.784        492.784        492.784          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.371          0.371          0.371          0.000
-- finished at 06/02/2020 22:26:52 --

mdtest-easy-stat
-- started at 06/02/2020 22:09:36 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '24000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-easy' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 14400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    1073896.694    1073892.532    1073895.986          0.403
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         13.409         13.409         13.409          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:09:49 --

mdtest-easy-write
-- started at 06/02/2020 21:26:58 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '24000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-easy' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 75.8%   Inodes: 858.3 Mi   Used Inodes: 4.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 14400000 files
Continue stonewall hit min: 233 max: 24000 avg: 15984.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      29360.606      29360.588      29360.594          0.004
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      31867.536             NA
   Tree creation             :         16.918         16.918         16.918          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        490.453        490.453        490.453          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.946             NA
   Tree creation             :          0.059          0.059          0.059          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 21:41:27 --

mdtest-hard-delete
-- started at 06/02/2020 22:27:42 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.2%   Inodes: 858.3 Mi   Used Inodes: 5.5%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      17791.361      17791.358      17791.360          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.655          0.655          0.655          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        269.726        269.726        269.726          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.526          1.526          1.526          0.000
-- finished at 06/02/2020 22:32:14 --

mdtest-hard-read
-- started at 06/02/2020 22:26:52 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.2%   Inodes: 858.3 Mi   Used Inodes: 5.5%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      95809.502      95797.847      95809.459          0.475
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         50.093         50.087         50.087          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:27:42 --

mdtest-hard-stat
-- started at 06/02/2020 22:18:05 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.3%   Inodes: 858.3 Mi   Used Inodes: 7.1%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     143148.156     143148.020     143148.098          0.024
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         33.523         33.523         33.523          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:18:38 --

mdtest-hard-write
-- started at 06/02/2020 21:49:50 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-19.41.54-app/mdtest-hard' '-x' './datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-19.41.54-app
FS: 50.0 TiB   Used FS: 85.2%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files
Continue stonewall hit min: 7794 max: 7998 avg: 7930.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      10945.819      10945.816      10945.817          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      15848.592             NA
   Tree creation             :        808.463        808.463        808.463          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        438.414        438.414        438.414          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.231             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 22:04:00 --

result
version         = io500-isc20
config-hash     = 8798F7A
result-dir      = ./results/2020.06.02-19.41.54-app
; START 2020-06-02 21:20:42


[ior-easy-write]
t_start         = 2020-06-02 21:20:42
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 7415.31
score           = 15.662637
t_delta         = 376.0656
t_end           = 2020-06-02 21:26:58

[mdtest-easy-write]
t_start         = 2020-06-02 21:26:58
exe             = ./mdtest -n 24000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-19.41.54-app/mdtest-easy -x ./datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 31.867536
score           = 29.360593
t_delta         = 869.6763
t_end           = 2020-06-02 21:41:27

[timestamp]
t_start         = 2020-06-02 21:41:27
t_delta         = 0.0010
t_end           = 2020-06-02 21:41:27

[ior-hard-write]
t_start         = 2020-06-02 21:41:27
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -w -D 300 -a POSIX --posix.odirect
throughput-stonewall = 98.32
score           = 9.454336
t_delta         = 502.4056
t_end           = 2020-06-02 21:49:50

[mdtest-hard-write]
t_start         = 2020-06-02 21:49:50
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 15.848592
score           = 10.945818
t_delta         = 850.5657
t_end           = 2020-06-02 22:04:00

[find]
t_start         = 2020-06-02 22:04:00
exe             = ./pfind ./datadir/2020.06.02-19.41.54-app -newer ./datadir/2020.06.02-19.41.54-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 128645
total-files     = 19199406
score           = 268.944060
t_delta         = 71.4426
t_end           = 2020-06-02 22:05:12

[ior-easy-read]
t_start         = 2020-06-02 22:05:12
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-easy/stonewall -t 32m -b 6000000m -F -r -R -a POSIX
score           = 22.365031
t_delta         = 264.0148
t_end           = 2020-06-02 22:09:36

[mdtest-easy-stat]
t_start         = 2020-06-02 22:09:36
exe             = ./mdtest -n 24000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-19.41.54-app/mdtest-easy -x ./datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall -T -a POSIX
score           = 1073.896694
t_delta         = 13.4482
t_end           = 2020-06-02 22:09:49

[ior-hard-read]
t_start         = 2020-06-02 22:09:49
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-19.41.54-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-19.41.54-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1800000 -r -R -a POSIX --posix.odirect
score           = 9.590026
t_delta         = 495.3104
t_end           = 2020-06-02 22:18:05

[mdtest-hard-stat]
t_start         = 2020-06-02 22:18:05
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -T -a POSIX
score           = 143.148130
t_delta         = 33.5448
t_end           = 2020-06-02 22:18:38

[mdtest-easy-delete]
t_start         = 2020-06-02 22:18:38
exe             = ./mdtest -n 24000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-19.41.54-app/mdtest-easy -x ./datadir/2020.06.02-19.41.54-app/mdtest-easy-stonewall -r -a POSIX
score           = 29.221732
t_delta         = 494.0693
t_end           = 2020-06-02 22:26:52

[mdtest-hard-read]
t_start         = 2020-06-02 22:26:52
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 95.809502
t_delta         = 50.1099
t_end           = 2020-06-02 22:27:42

[mdtest-hard-delete]
t_start         = 2020-06-02 22:27:42
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-19.41.54-app/mdtest-hard -x ./datadir/2020.06.02-19.41.54-app/mdtest-hard-stonewall -r -a POSIX
score           = 17.791361
t_delta         = 271.3160
t_end           = 2020-06-02 22:32:14

[SCORE]
MD              = 71.218588
BW              = 13.349694
SCORE           = 30.834175 
hash            = 17F1E90D
; END 2020-06-02 22:32:18
result_summary
IO500 version io500-isc20
[RESULT]       ior-easy-write       15.662637 GiB/s  : time 376.066 seconds
[RESULT]    mdtest-easy-write       29.360593 kIOPS : time 869.676 seconds
[RESULT]       ior-hard-write        9.454336 GiB/s  : time 502.406 seconds
[RESULT]    mdtest-hard-write       10.945818 kIOPS : time 850.566 seconds
[RESULT]                 find      268.944060 kIOPS : time 71.443 seconds
[RESULT]        ior-easy-read       22.365031 GiB/s  : time 264.015 seconds
[RESULT]     mdtest-easy-stat     1073.896694 kIOPS : time 13.448 seconds
[RESULT]        ior-hard-read        9.590026 GiB/s  : time 495.310 seconds
[RESULT]     mdtest-hard-stat      143.148130 kIOPS : time 33.545 seconds
[RESULT]   mdtest-easy-delete       29.221732 kIOPS : time 494.069 seconds
[RESULT]     mdtest-hard-read       95.809502 kIOPS : time 50.110 seconds
[RESULT]   mdtest-hard-delete       17.791361 kIOPS : time 271.316 seconds
[SCORE] Bandwidth 13.349694 GB/s : IOPS 71.218588 kiops : TOTAL 30.834175