SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.7.0
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 600
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 20
Data Storage Devices 80

METADATA

Easy Write 33.73 kIOP/s
Easy Stat 1,088.99 kIOP/s
Easy Delete 32.47 kIOP/s
Hard Write 14.51 kIOP/s
Hard Read 91.44 kIOP/s
Hard Stat 147.39 kIOP/s
Hard Delete 17.36 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 11:08:32 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 11:08:32 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-easy
FS                  : 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 5676.27 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 6241124352000000.
WARNING: Stat() of aggregate file size      = 9691358822400.
WARNING: Using actual aggregate bytes moved = 9691358822400.
read      23128      5782       0.000161    10158080000 4096       0.008367   399.59     0.008124   399.61     0   
Max Read:  23128.49 MiB/sec (24251.98 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        23128.49   23128.49   23128.49       0.00    5782.12    5782.12    5782.12       0.00  399.61107         NA            NA     0    600  60    1   1     1        1         0    0      1 10401873920000  4194304 9242400.0 POSIX      0
Finished            : Tue Jun  2 11:15:12 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 10:30:58 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 10:30:58 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-easy
FS                  : 50.0 TiB   Used FS: 43.8%   Inodes: 858.3 Mi   Used Inodes: 2.5%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 5676.27 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1536 max: 3851 -- min data: 6.0 GiB mean data: 8.1 GiB time: 301.0s
WARNING: Expected aggregate file size       = 6241124352000000.
WARNING: Stat() of aggregate file size      = 9691358822400.
WARNING: Using actual aggregate bytes moved = 9691358822400.
WARNING: maybe caused by deadlineForStonewalling
write     11995      2999.15    0.000280    10158080000 4096       0.074995   770.42     0.019409   770.51     0   
Max Write: 11995.13 MiB/sec (12577.81 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       11995.13   11995.13   11995.13       0.00    2998.78    2998.78    2998.78       0.00  770.51253     301.04      16440.98     0    600  60    1   1     1        1         0    0      1 10401873920000  4194304 9242400.0 POSIX      0
Finished            : Tue Jun  2 10:43:48 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 11:15:28 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -r -R -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 11:15:28 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-hard
FS                  : 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 48.74 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 162258 max: 162258 -- min data: 7.1 GiB mean data: 7.1 GiB time: 437.5s
WARNING: Expected aggregate file size       = 53589120000000.
WARNING: Stat() of aggregate file size      = 4576454438400.
WARNING: Using actual aggregate bytes moved = 4576454438400.
read      9974       222500     437.55      45.91      45.91      0.011708   437.55     0.000575   437.56     0   
Max Read:  9974.49 MiB/sec (10459.01 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         9974.49    9974.49    9974.49       0.00  222494.27  222494.27  222494.27       0.00  437.56094     437.48       9976.28     0    600  60    1   0     1        1         0    0 1900000    47008    47008 4364447.0 POSIX      0
Finished            : Tue Jun  2 11:22:46 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 10:53:03 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -w -D 300 -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 10:53:03 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-hard
FS                  : 50.0 TiB   Used FS: 61.6%   Inodes: 858.3 Mi   Used Inodes: 5.5%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 48.74 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 18 max: 162258 -- min data: 0.0 GiB mean data: 0.1 GiB time: 300.1s
WARNING: Expected aggregate file size       = 53589120000000.
WARNING: Stat() of aggregate file size      = 4576454438400.
WARNING: Using actual aggregate bytes moved = 4576454438400.
WARNING: maybe caused by deadlineForStonewalling
write     9214       205528     463.49      45.91      45.91      0.002429   473.68     0.000319   473.68     0   
Max Write: 9213.84 MiB/sec (9661.42 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        9213.84    9213.84    9213.84       0.00  205527.06  205527.06  205527.06       0.00  473.68361     300.07        248.15     0    600  60    1   0     1        1         0    0 1900000    47008    47008 4364447.0 POSIX      0
Finished            : Tue Jun  2 11:00:57 2020
mdtest-easy-delete
-- started at 06/02/2020 11:23:18 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '29000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-easy' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 17400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      32469.439      32469.427      32469.433          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.695          2.695          2.695          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        535.889        535.889        535.889          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.371          0.371          0.371          0.000
-- finished at 06/02/2020 11:32:15 --

mdtest-easy-stat
-- started at 06/02/2020 11:15:12 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '29000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-easy' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 17400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    1089045.689    1088441.710    1088984.367         24.232
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         15.986         15.977         15.978          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:15:28 --

mdtest-easy-write
-- started at 06/02/2020 10:43:48 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '29000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-easy' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 61.5%   Inodes: 858.3 Mi   Used Inodes: 3.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 17400000 files
Continue stonewall hit min: 563 max: 29000 avg: 17805.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      33730.228      33730.215      33730.222          0.003
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      35500.400             NA
   Tree creation             :         16.387         16.387         16.387          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        515.858        515.858        515.858          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.939             NA
   Tree creation             :          0.061          0.061          0.061          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 10:53:03 --

mdtest-hard-delete
-- started at 06/02/2020 11:33:07 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 69.9%   Inodes: 858.3 Mi   Used Inodes: 4.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      17359.863      17359.858      17359.861          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.713          0.713          0.713          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        273.424        273.424        273.424          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.403          1.403          1.403          0.000
-- finished at 06/02/2020 11:37:42 --

mdtest-hard-read
-- started at 06/02/2020 11:32:15 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 69.9%   Inodes: 858.3 Mi   Used Inodes: 4.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      91437.934      91437.847      91437.894          0.020
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         51.911         51.911         51.911          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:33:07 --

mdtest-hard-stat
-- started at 06/02/2020 11:22:46 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     147392.859     147392.109     147392.214          0.063
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         32.204         32.204         32.204          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:23:18 --

mdtest-hard-write
-- started at 06/02/2020 11:00:57 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 69.9%   Inodes: 858.3 Mi   Used Inodes: 6.0%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files
Continue stonewall hit min: 7762 max: 7911 avg: 7843.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      14508.550      14508.541      14508.546          0.002
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      15683.956             NA
   Tree creation             :        860.547        860.547        860.547          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        327.159        327.159        327.159          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.043             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:07:38 --

result
version         = io500-isc20
config-hash     = 8299AA79
result-dir      = ./results/2020.06.02-08.56.31-app
; START 2020-06-02 10:30:58


[ior-easy-write]
t_start         = 2020-06-02 10:30:58
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 3763.75
score           = 11.713997
t_delta         = 770.6032
t_end           = 2020-06-02 10:43:48

[mdtest-easy-write]
t_start         = 2020-06-02 10:43:48
exe             = ./mdtest -n 29000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-08.56.31-app/mdtest-easy -x ./datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 35.500400
score           = 33.730224
t_delta         = 554.6556
t_end           = 2020-06-02 10:53:03

[timestamp]
t_start         = 2020-06-02 10:53:03
t_delta         = 0.0011
t_end           = 2020-06-02 10:53:03

[ior-hard-write]
t_start         = 2020-06-02 10:53:03
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -w -D 300 -a POSIX --posix.odirect
throughput-stonewall = 92.11
score           = 8.997895
t_delta         = 473.6986
t_end           = 2020-06-02 11:00:57

[mdtest-hard-write]
t_start         = 2020-06-02 11:00:57
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 15.683956
score           = 14.508545
t_delta         = 401.3126
t_end           = 2020-06-02 11:07:38

[find]
t_start         = 2020-06-02 11:07:38
exe             = ./pfind ./datadir/2020.06.02-08.56.31-app -newer ./datadir/2020.06.02-08.56.31-app/timestampfile -size 3901c -name *01* -C -N -q 20000
found           = 128210
total-files     = 22147206
score           = 407.596497
t_delta         = 54.4314
t_end           = 2020-06-02 11:08:32

[ior-easy-read]
t_start         = 2020-06-02 11:08:32
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
score           = 22.586415
t_delta         = 399.7544
t_end           = 2020-06-02 11:15:12

[mdtest-easy-stat]
t_start         = 2020-06-02 11:15:12
exe             = ./mdtest -n 29000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-08.56.31-app/mdtest-easy -x ./datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall -T -a POSIX
score           = 1088.986733
t_delta         = 16.0647
t_end           = 2020-06-02 11:15:28

[ior-hard-read]
t_start         = 2020-06-02 11:15:28
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -r -R -a POSIX --posix.odirect
score           = 9.740713
t_delta         = 437.6888
t_end           = 2020-06-02 11:22:46

[mdtest-hard-stat]
t_start         = 2020-06-02 11:22:46
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -T -a POSIX
score           = 147.392227
t_delta         = 32.2236
t_end           = 2020-06-02 11:23:18

[mdtest-easy-delete]
t_start         = 2020-06-02 11:23:18
exe             = ./mdtest -n 29000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-08.56.31-app/mdtest-easy -x ./datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall -r -a POSIX
score           = 32.469434
t_delta         = 537.1731
t_end           = 2020-06-02 11:32:15

[mdtest-hard-read]
t_start         = 2020-06-02 11:32:15
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 91.437934
t_delta         = 52.0593
t_end           = 2020-06-02 11:33:07

[mdtest-hard-delete]
t_start         = 2020-06-02 11:33:07
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -r -a POSIX
score           = 17.359863
t_delta         = 274.8469
t_end           = 2020-06-02 11:37:42

[SCORE]
MD              = 79.834344
BW              = 12.340166
SCORE           = 31.387403 
hash            = 17E03746
; END 2020-06-02 11:37:47
result_summary
IO500 version io500-isc20
[RESULT]       ior-easy-write       11.713997 GiB/s  : time 770.603 seconds
[RESULT]    mdtest-easy-write       33.730224 kIOPS : time 554.656 seconds
[RESULT]       ior-hard-write        8.997895 GiB/s  : time 473.699 seconds
[RESULT]    mdtest-hard-write       14.508545 kIOPS : time 401.313 seconds
[RESULT]                 find      407.596497 kIOPS : time 54.431 seconds
[RESULT]        ior-easy-read       22.586415 GiB/s  : time 399.754 seconds
[RESULT]     mdtest-easy-stat     1088.986733 kIOPS : time 16.065 seconds
[RESULT]        ior-hard-read        9.740713 GiB/s  : time 437.689 seconds
[RESULT]     mdtest-hard-stat      147.392227 kIOPS : time 32.224 seconds
[RESULT]   mdtest-easy-delete       32.469434 kIOPS : time 537.173 seconds
[RESULT]     mdtest-hard-read       91.437934 kIOPS : time 52.059 seconds
[RESULT]   mdtest-hard-delete       17.359863 kIOPS : time 274.847 seconds
[SCORE] Bandwidth 12.340166 GB/s : IOPS 79.834344 kiops : TOTAL 31.387403