SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.8.1
OS Version 7.7

INFORMATION

Client Nodes 40
Client Total Procs 1,040
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 20
Data Storage Devices 80

METADATA

Easy Write 289.75 kIOP/s
Easy Stat 3,435.11 kIOP/s
Easy Delete 489.99 kIOP/s
Hard Write 50.16 kIOP/s
Hard Read 329.95 kIOP/s
Hard Stat 469.73 kIOP/s
Hard Delete 65.70 kIOP/s

Submitted Files

ior-easy-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Oct 13 21:14:12 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-easy.stonewall -t 32m -b 9920000m -F -r -R -a POSIX
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Tue Oct 13 21:14:12 2020
Path                : ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy.00000000
FS                  : 150.0 TiB   Used FS: 30.6%   Inodes: 858.3 Mi   Used Inodes: 19.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1040
clients per node    : 26
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 9838.87 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 10817948876800000
WARNING: Stat() of aggregate file size      = 36187783823360
WARNING: Using actual aggregate bytes moved = 36187783823360
read      77043      2407.62    0.001445    10158080000 32768      0.005263   447.94     0.002478   447.95     0   
ior-easy-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Oct 13 20:35:18 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-easy.stonewall -t 32m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Tue Oct 13 20:35:18 2020
Path                : ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy.00000000
FS                  : 150.0 TiB   Used FS: 0.0%   Inodes: 858.3 Mi   Used Inodes: 0.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1040
clients per node    : 26
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 9838.87 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 557 max: 1037 -- min data: 17.4 GiB mean data: 23.9 GiB time: 302.3s
WARNING: Expected aggregate file size       = 10817948876800000
WARNING: Stat() of aggregate file size      = 36187783823360
WARNING: Using actual aggregate bytes moved = 36187783823360
WARNING: Maybe caused by deadlineForStonewalling
write     77393      2418.78    0.001362    10158080000 32768      0.042550   445.88     0.002954   445.92     0   
ior-hard-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Oct 13 21:22:12 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.13-20.35.18/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX --posix.odirect=TRUE
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Tue Oct 13 21:22:12 2020
Path                : ./datadir/2020.10.13-20.35.18/ior-hard/file
FS                  : 150.0 TiB   Used FS: 30.6%   Inodes: 858.3 Mi   Used Inodes: 19.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.13-20.35.18/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1040
clients per node    : 26
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 444.64 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 275946 max: 275946 -- min data: 12.1 GiB mean data: 12.1 GiB time: 347.3s
WARNING: Expected aggregate file size       = 488883200000000
WARNING: Stat() of aggregate file size      = 13490536350720
WARNING: Using actual aggregate bytes moved = 13490536350720
read      37006      825489     347.63      45.91      45.91      0.005567   347.65     0.002212   347.66     0   
ior-hard-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Oct 13 20:51:10 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.13-20.35.18/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX --posix.odirect=TRUE
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Tue Oct 13 20:51:10 2020
Path                : ./datadir/2020.10.13-20.35.18/ior-hard/file
FS                  : 150.0 TiB   Used FS: 22.3%   Inodes: 858.3 Mi   Used Inodes: 16.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.13-20.35.18/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1040
clients per node    : 26
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 444.64 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 19 max: 275946 -- min data: 0.0 GiB mean data: 0.5 GiB time: 300.5s
WARNING: Expected aggregate file size       = 488883200000000
WARNING: Stat() of aggregate file size      = 13490536350720
WARNING: Using actual aggregate bytes moved = 13490536350720
WARNING: Maybe caused by deadlineForStonewalling
write     18051      402664     669.50      45.91      45.91      0.009886   712.71     0.001852   712.72     0   
mdtest-easy-delete
-- started at 10/13/2020 21:28:39 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.13-20.35.18/mdtest-easy' '-x' './results/2020.10.13-20.35.18/mdtest-easy.stonewall' '-r' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 30.6%   Inodes: 858.3 Mi   Used Inodes: 19.4%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     489987.523     489986.379     489987.186          0.197
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.254          1.254          1.254          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        218.438        218.437        218.438          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.798          0.798          0.798          0.000
-- finished at 10/13/2020 21:32:19 --

mdtest-easy-stat
-- started at 10/13/2020 21:21:40 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.13-20.35.18/mdtest-easy' '-x' './results/2020.10.13-20.35.18/mdtest-easy.stonewall' '-T' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 30.6%   Inodes: 858.3 Mi   Used Inodes: 19.4%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    3435109.776    3435097.974    3435103.660          2.451
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         31.158         31.158         31.158          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/13/2020 21:22:11 --

mdtest-easy-write
-- started at 10/13/2020 20:42:44 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.13-20.35.18/mdtest-easy' '-x' './results/2020.10.13-20.35.18/mdtest-easy.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 22.0%   Inodes: 858.3 Mi   Used Inodes: 3.8%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files
Continue stonewall hit min: 86734 max: 102915 avg: 96458.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     289748.966     289748.493     289748.717          0.107
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     334381.652             NA
   Tree creation             :         23.693         23.693         23.693          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        369.395        369.394        369.395          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.007             NA
   Tree creation             :          0.042          0.042          0.042          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/13/2020 20:51:10 --

mdtest-hard-delete
-- started at 10/13/2020 21:33:16 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.13-20.35.18/mdtest-hard' '-x' './results/2020.10.13-20.35.18/mdtest-hard.stonewall' '-r' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 30.3%   Inodes: 858.3 Mi   Used Inodes: 7.3%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      65697.338      65697.082      65697.284          0.038
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.358          0.358          0.358          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        273.182        273.181        273.181          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.796          2.796          2.796          0.000
-- finished at 10/13/2020 21:37:53 --

mdtest-hard-read
-- started at 10/13/2020 21:32:19 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.13-20.35.18/mdtest-hard' '-x' './results/2020.10.13-20.35.18/mdtest-hard.stonewall' '-E' '-X' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 30.3%   Inodes: 858.3 Mi   Used Inodes: 7.3%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     329946.880     329944.167     329946.281          0.398
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         54.395         54.394         54.395          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/13/2020 21:33:15 --

mdtest-hard-stat
-- started at 10/13/2020 21:28:01 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.13-20.35.18/mdtest-hard' '-x' './results/2020.10.13-20.35.18/mdtest-hard.stonewall' '-T' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 30.6%   Inodes: 858.3 Mi   Used Inodes: 19.4%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     469734.761     469733.703     469734.197          0.220
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         38.207         38.207         38.207          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/13/2020 21:28:39 --

mdtest-hard-write
-- started at 10/13/2020 21:03:02 --

mdtest-3.4.0+dev was launched with 1040 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.13-20.35.18/mdtest-hard' '-x' './results/2020.10.13-20.35.18/mdtest-hard.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.13-20.35.18
FS: 150.0 TiB   Used FS: 30.5%   Inodes: 858.3 Mi   Used Inodes: 17.4%

Nodemap: 11111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1040 tasks, 1040000000 files
Continue stonewall hit min: 15966 max: 17257 avg: 16599.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      50162.887      50162.626      50162.826          0.037
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      57540.176             NA
   Tree creation             :        786.776        786.776        786.776          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        357.782        357.780        357.780          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.021             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/13/2020 21:11:52 --

result
version         = io500-sc20
config-hash     = 16684A95
result-dir      = ./results/2020.10.13-20.35.18
; START 2020-10-13 20:35:18


[ior-easy-write]
t_start         = 2020-10-13 20:35:18
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-easy.stonewall -t 32m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 58066.77
score           = 75.579211
t_delta         = 446.0466
t_end           = 2020-10-13 20:42:44

[mdtest-easy-write]
t_start         = 2020-10-13 20:42:44
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.13-20.35.18/mdtest-easy -x ./results/2020.10.13-20.35.18/mdtest-easy.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 334.381652
score           = 289.748670
t_delta         = 505.3250
t_end           = 2020-10-13 20:51:10

[timestamp]
t_start         = 2020-10-13 20:51:10
t_delta         = 0.0012
t_end           = 2020-10-13 20:51:10

[ior-hard-write]
t_start         = 2020-10-13 20:51:10
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.13-20.35.18/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX --posix.odirect=TRUE
throughput-stonewall = 720.40
score           = 17.628231
t_delta         = 712.7429
t_end           = 2020-10-13 21:03:02

[mdtest-hard-write]
t_start         = 2020-10-13 21:03:02
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.13-20.35.18/mdtest-hard -x ./results/2020.10.13-20.35.18/mdtest-hard.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 57.540176
score           = 50.162840
t_delta         = 529.5017
t_end           = 2020-10-13 21:11:52

[find]
t_start         = 2020-10-13 21:11:52
exe             = ./pfind ./datadir/2020.10.13-20.35.18 -newer ./results/2020.10.13-20.35.18/timestampfile -size 3901c -name *01* -C -q 10000
found           = 794960
total-files     = 124979921
score           = 892.413986
t_delta         = 140.0797
t_end           = 2020-10-13 21:14:12

[ior-easy-read]
t_start         = 2020-10-13 21:14:12
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.13-20.35.18/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-easy.stonewall -t 32m -b 9920000m -F -r -R -a POSIX
score           = 75.237432
t_delta         = 448.0087
t_end           = 2020-10-13 21:21:40

[mdtest-easy-stat]
t_start         = 2020-10-13 21:21:40
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.13-20.35.18/mdtest-easy -x ./results/2020.10.13-20.35.18/mdtest-easy.stonewall -T -a POSIX
score           = 3435.108094
t_delta         = 31.1893
t_end           = 2020-10-13 21:22:11

[ior-hard-read]
t_start         = 2020-10-13 21:22:11
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.13-20.35.18/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.13-20.35.18/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX --posix.odirect=TRUE
score           = 36.139061
t_delta         = 348.6890
t_end           = 2020-10-13 21:28:00

[mdtest-hard-stat]
t_start         = 2020-10-13 21:28:00
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.13-20.35.18/mdtest-hard -x ./results/2020.10.13-20.35.18/mdtest-hard.stonewall -T -a POSIX
score           = 469.734184
t_delta         = 39.2383
t_end           = 2020-10-13 21:28:39

[mdtest-easy-delete]
t_start         = 2020-10-13 21:28:39
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.13-20.35.18/mdtest-easy -x ./results/2020.10.13-20.35.18/mdtest-easy.stonewall -r -a POSIX
score           = 489.987314
t_delta         = 220.1139
t_end           = 2020-10-13 21:32:19

[mdtest-hard-read]
t_start         = 2020-10-13 21:32:19
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.13-20.35.18/mdtest-hard -x ./results/2020.10.13-20.35.18/mdtest-hard.stonewall -E -X -a POSIX
score           = 329.946655
t_delta         = 55.5482
t_end           = 2020-10-13 21:33:15

[mdtest-hard-delete]
t_start         = 2020-10-13 21:33:15
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.13-20.35.18/mdtest-hard -x ./results/2020.10.13-20.35.18/mdtest-hard.stonewall -r -a POSIX
score           = 65.697296
t_delta         = 278.0114
t_end           = 2020-10-13 21:37:53

[SCORE]
MD              = 349.435739
BW              = 43.627013
SCORE           = 123.469986 
hash            = DF66D45B
; END 2020-10-13 21:37:57
result_summary
IO500 version io500-sc20
[RESULT]       ior-easy-write       75.579211 GiB/s : time 446.047 seconds
[RESULT]    mdtest-easy-write      289.748670 kIOPS : time 505.325 seconds
[RESULT]       ior-hard-write       17.628231 GiB/s : time 712.743 seconds
[RESULT]    mdtest-hard-write       50.162840 kIOPS : time 529.502 seconds
[RESULT]                 find      892.413986 kIOPS : time 140.080 seconds
[RESULT]        ior-easy-read       75.237432 GiB/s : time 448.009 seconds
[RESULT]     mdtest-easy-stat     3435.108094 kIOPS : time 31.189 seconds
[RESULT]        ior-hard-read       36.139061 GiB/s : time 348.689 seconds
[RESULT]     mdtest-hard-stat      469.734184 kIOPS : time 39.238 seconds
[RESULT]   mdtest-easy-delete      489.987314 kIOPS : time 220.114 seconds
[RESULT]     mdtest-hard-read      329.946655 kIOPS : time 55.548 seconds
[RESULT]   mdtest-hard-delete       65.697296 kIOPS : time 278.011 seconds
[SCORE] Bandwidth 43.627013 GB/s : IOPS 349.435739 kiops : TOTAL 123.469986