SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.7.0
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 600

METADATA

Easy Write 33.73 kIOP/s
Easy Stat 1,088.99 kIOP/s
Easy Delete 32.47 kIOP/s
Hard Write 14.51 kIOP/s
Hard Read 91.44 kIOP/s
Hard Stat 147.39 kIOP/s
Hard Delete 17.36 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 11:08:32 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 11:08:32 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-easy
FS                  : 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 5676.27 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 6241124352000000.
WARNING: Stat() of aggregate file size      = 9691358822400.
WARNING: Using actual aggregate bytes moved = 9691358822400.
read      23128      5782       0.000161    10158080000 4096       0.008367   399.59     0.008124   399.61     0   
Max Read:  23128.49 MiB/sec (24251.98 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        23128.49   23128.49   23128.49       0.00    5782.12    5782.12    5782.12       0.00  399.61107         NA            NA     0    600  60    1   1     1        1         0    0      1 10401873920000  4194304 9242400.0 POSIX      0
Finished            : Tue Jun  2 11:15:12 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 10:30:58 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 10:30:58 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-easy
FS                  : 50.0 TiB   Used FS: 43.8%   Inodes: 858.3 Mi   Used Inodes: 2.5%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 5676.27 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1536 max: 3851 -- min data: 6.0 GiB mean data: 8.1 GiB time: 301.0s
WARNING: Expected aggregate file size       = 6241124352000000.
WARNING: Stat() of aggregate file size      = 9691358822400.
WARNING: Using actual aggregate bytes moved = 9691358822400.
WARNING: maybe caused by deadlineForStonewalling
write     11995      2999.15    0.000280    10158080000 4096       0.074995   770.42     0.019409   770.51     0   
Max Write: 11995.13 MiB/sec (12577.81 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       11995.13   11995.13   11995.13       0.00    2998.78    2998.78    2998.78       0.00  770.51253     301.04      16440.98     0    600  60    1   1     1        1         0    0      1 10401873920000  4194304 9242400.0 POSIX      0
Finished            : Tue Jun  2 10:43:48 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 11:15:28 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -r -R -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 11:15:28 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-hard
FS                  : 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 48.74 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 162258 max: 162258 -- min data: 7.1 GiB mean data: 7.1 GiB time: 437.5s
WARNING: Expected aggregate file size       = 53589120000000.
WARNING: Stat() of aggregate file size      = 4576454438400.
WARNING: Using actual aggregate bytes moved = 4576454438400.
read      9974       222500     437.55      45.91      45.91      0.011708   437.55     0.000575   437.56     0   
Max Read:  9974.49 MiB/sec (10459.01 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         9974.49    9974.49    9974.49       0.00  222494.27  222494.27  222494.27       0.00  437.56094     437.48       9976.28     0    600  60    1   0     1        1         0    0 1900000    47008    47008 4364447.0 POSIX      0
Finished            : Tue Jun  2 11:22:46 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jun  2 10:53:03 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -w -D 300 -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Tue Jun  2 10:53:03 2020
Path                : /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app/ior-hard
FS                  : 50.0 TiB   Used FS: 61.6%   Inodes: 858.3 Mi   Used Inodes: 5.5%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.06.02-08.56.31-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 600
clients per node    : 60
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 48.74 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 18 max: 162258 -- min data: 0.0 GiB mean data: 0.1 GiB time: 300.1s
WARNING: Expected aggregate file size       = 53589120000000.
WARNING: Stat() of aggregate file size      = 4576454438400.
WARNING: Using actual aggregate bytes moved = 4576454438400.
WARNING: maybe caused by deadlineForStonewalling
write     9214       205528     463.49      45.91      45.91      0.002429   473.68     0.000319   473.68     0   
Max Write: 9213.84 MiB/sec (9661.42 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        9213.84    9213.84    9213.84       0.00  205527.06  205527.06  205527.06       0.00  473.68361     300.07        248.15     0    600  60    1   0     1        1         0    0 1900000    47008    47008 4364447.0 POSIX      0
Finished            : Tue Jun  2 11:00:57 2020
mdtest-easy-delete
-- started at 06/02/2020 11:23:18 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '29000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-easy' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 17400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      32469.439      32469.427      32469.433          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.695          2.695          2.695          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        535.889        535.889        535.889          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.371          0.371          0.371          0.000
-- finished at 06/02/2020 11:32:15 --

mdtest-easy-stat
-- started at 06/02/2020 11:15:12 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '29000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-easy' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 17400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    1089045.689    1088441.710    1088984.367         24.232
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         15.986         15.977         15.978          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:15:28 --

mdtest-easy-write
-- started at 06/02/2020 10:43:48 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '29000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-easy' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 61.5%   Inodes: 858.3 Mi   Used Inodes: 3.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 17400000 files
Continue stonewall hit min: 563 max: 29000 avg: 17805.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      33730.228      33730.215      33730.222          0.003
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      35500.400             NA
   Tree creation             :         16.387         16.387         16.387          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        515.858        515.858        515.858          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.939             NA
   Tree creation             :          0.061          0.061          0.061          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 10:53:03 --

mdtest-hard-delete
-- started at 06/02/2020 11:33:07 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 69.9%   Inodes: 858.3 Mi   Used Inodes: 4.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      17359.863      17359.858      17359.861          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.713          0.713          0.713          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        273.424        273.424        273.424          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.403          1.403          1.403          0.000
-- finished at 06/02/2020 11:37:42 --

mdtest-hard-read
-- started at 06/02/2020 11:32:15 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 69.9%   Inodes: 858.3 Mi   Used Inodes: 4.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      91437.934      91437.847      91437.894          0.020
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         51.911         51.911         51.911          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:33:07 --

mdtest-hard-stat
-- started at 06/02/2020 11:22:46 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 70.0%   Inodes: 858.3 Mi   Used Inodes: 6.6%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     147392.859     147392.109     147392.214          0.063
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         32.204         32.204         32.204          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:23:18 --

mdtest-hard-write
-- started at 06/02/2020 11:00:57 --

mdtest-3.3.0+dev was launched with 600 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.06.02-08.56.31-app/mdtest-hard' '-x' './datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /io500/io500-isc20/io500-app/datadir/2020.06.02-08.56.31-app
FS: 50.0 TiB   Used FS: 69.9%   Inodes: 858.3 Mi   Used Inodes: 6.0%

Nodemap: 111111111111111111111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 60 for each phase.
600 tasks, 600000000 files
Continue stonewall hit min: 7762 max: 7911 avg: 7843.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      14508.550      14508.541      14508.546          0.002
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      15683.956             NA
   Tree creation             :        860.547        860.547        860.547          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        327.159        327.159        327.159          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.043             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/02/2020 11:07:38 --

result
version         = io500-isc20
config-hash     = 8299AA79
result-dir      = ./results/2020.06.02-08.56.31-app
; START 2020-06-02 10:30:58


[ior-easy-write]
t_start         = 2020-06-02 10:30:58
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 3763.75
score           = 11.713997
t_delta         = 770.6032
t_end           = 2020-06-02 10:43:48

[mdtest-easy-write]
t_start         = 2020-06-02 10:43:48
exe             = ./mdtest -n 29000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-08.56.31-app/mdtest-easy -x ./datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 35.500400
score           = 33.730224
t_delta         = 554.6556
t_end           = 2020-06-02 10:53:03

[timestamp]
t_start         = 2020-06-02 10:53:03
t_delta         = 0.0011
t_end           = 2020-06-02 10:53:03

[ior-hard-write]
t_start         = 2020-06-02 10:53:03
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -w -D 300 -a POSIX --posix.odirect
throughput-stonewall = 92.11
score           = 8.997895
t_delta         = 473.6986
t_end           = 2020-06-02 11:00:57

[mdtest-hard-write]
t_start         = 2020-06-02 11:00:57
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 15.683956
score           = 14.508545
t_delta         = 401.3126
t_end           = 2020-06-02 11:07:38

[find]
t_start         = 2020-06-02 11:07:38
exe             = ./pfind ./datadir/2020.06.02-08.56.31-app -newer ./datadir/2020.06.02-08.56.31-app/timestampfile -size 3901c -name *01* -C -N -q 20000
found           = 128210
total-files     = 22147206
score           = 407.596497
t_delta         = 54.4314
t_end           = 2020-06-02 11:08:32

[ior-easy-read]
t_start         = 2020-06-02 11:08:32
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
score           = 22.586415
t_delta         = 399.7544
t_end           = 2020-06-02 11:15:12

[mdtest-easy-stat]
t_start         = 2020-06-02 11:15:12
exe             = ./mdtest -n 29000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-08.56.31-app/mdtest-easy -x ./datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall -T -a POSIX
score           = 1088.986733
t_delta         = 16.0647
t_end           = 2020-06-02 11:15:28

[ior-hard-read]
t_start         = 2020-06-02 11:15:28
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.06.02-08.56.31-app/ior-hard/file -O stoneWallingStatusFile=./datadir/2020.06.02-08.56.31-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1900000 -r -R -a POSIX --posix.odirect
score           = 9.740713
t_delta         = 437.6888
t_end           = 2020-06-02 11:22:46

[mdtest-hard-stat]
t_start         = 2020-06-02 11:22:46
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -T -a POSIX
score           = 147.392227
t_delta         = 32.2236
t_end           = 2020-06-02 11:23:18

[mdtest-easy-delete]
t_start         = 2020-06-02 11:23:18
exe             = ./mdtest -n 29000 -u -L -F -P -N 1 -d ./datadir/2020.06.02-08.56.31-app/mdtest-easy -x ./datadir/2020.06.02-08.56.31-app/mdtest-easy-stonewall -r -a POSIX
score           = 32.469434
t_delta         = 537.1731
t_end           = 2020-06-02 11:32:15

[mdtest-hard-read]
t_start         = 2020-06-02 11:32:15
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 91.437934
t_delta         = 52.0593
t_end           = 2020-06-02 11:33:07

[mdtest-hard-delete]
t_start         = 2020-06-02 11:33:07
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.06.02-08.56.31-app/mdtest-hard -x ./datadir/2020.06.02-08.56.31-app/mdtest-hard-stonewall -r -a POSIX
score           = 17.359863
t_delta         = 274.8469
t_end           = 2020-06-02 11:37:42

[SCORE]
MD              = 79.834344
BW              = 12.340166
SCORE           = 31.387403 
hash            = 17E03746
; END 2020-06-02 11:37:47
result_summary
IO500 version io500-isc20
[RESULT]       ior-easy-write       11.713997 GiB/s  : time 770.603 seconds
[RESULT]    mdtest-easy-write       33.730224 kIOPS : time 554.656 seconds
[RESULT]       ior-hard-write        8.997895 GiB/s  : time 473.699 seconds
[RESULT]    mdtest-hard-write       14.508545 kIOPS : time 401.313 seconds
[RESULT]                 find      407.596497 kIOPS : time 54.431 seconds
[RESULT]        ior-easy-read       22.586415 GiB/s  : time 399.754 seconds
[RESULT]     mdtest-easy-stat     1088.986733 kIOPS : time 16.065 seconds
[RESULT]        ior-hard-read        9.740713 GiB/s  : time 437.689 seconds
[RESULT]     mdtest-hard-stat      147.392227 kIOPS : time 32.224 seconds
[RESULT]   mdtest-easy-delete       32.469434 kIOPS : time 537.173 seconds
[RESULT]     mdtest-hard-read       91.437934 kIOPS : time 52.059 seconds
[RESULT]   mdtest-hard-delete       17.359863 kIOPS : time 274.847 seconds
[SCORE] Bandwidth 12.340166 GB/s : IOPS 79.834344 kiops : TOTAL 31.387403