SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.9.0
OS Version 7.7

INFORMATION

Client Nodes 48
Client Total Procs 1,248

METADATA

Easy Write 301.38 kIOP/s
Easy Stat 4,314.48 kIOP/s
Easy Delete 451.13 kIOP/s
Hard Write 69.44 kIOP/s
Hard Read 327.24 kIOP/s
Hard Stat 506.15 kIOP/s
Hard Delete 81.63 kIOP/s

Submitted Files

io500
#!/bin/bash

export I_MPI_FABRICS=shm:dapl
export I_MPI_FALLBACK=tcp
export I_MPI_DEBUG=0
export I_MPI_ROOT=/misc/local/INTEL-2018/compilers_and_libraries_2018.0.128/linux/mpi
export DAT_OVERRIDE=/misc/local/admin/cx5_dat.conf

mpirun -np 1248 -ppn 26 -f hostfile-48-mix ./io500 config-hhmi.ini

ior-easy-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 14:56:34 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 14:56:34 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy.00000000
FS                  : 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 11806.64 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 12981538652160000
WARNING: Stat() of aggregate file size      = 39949638303744
WARNING: Using actual aggregate bytes moved = 39949638303744
read      82607      2581.50    0.001488    10158080000 32768      0.006506   461.20     0.003419   461.21     0   
ior-easy-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 14:18:58 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 14:18:58 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy.00000000
FS                  : 150.0 TiB   Used FS: 25.4%   Inodes: 858.3 Mi   Used Inodes: 26.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 11806.64 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 397 max: 954 -- min data: 12.4 GiB mean data: 19.8 GiB time: 300.4s
WARNING: Expected aggregate file size       = 12981538652160000
WARNING: Stat() of aggregate file size      = 39949638303744
WARNING: Using actual aggregate bytes moved = 39949638303744
WARNING: Maybe caused by deadlineForStonewalling
write     77155      2411.39    0.001476    10158080000 32768      0.054248   493.74     0.004369   493.79     0   
ior-hard-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 15:04:40 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 15:04:40 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-hard/file
FS                  : 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 533.56 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 298372 max: 298372 -- min data: 13.1 GiB mean data: 13.1 GiB time: 367.3s
WARNING: Expected aggregate file size       = 586659840000000
WARNING: Stat() of aggregate file size      = 17504286978048
WARNING: Using actual aggregate bytes moved = 17504286978048
read      45417      1013118    367.54      45.91      45.91      0.014385   367.55     0.003304   367.56     0   
ior-hard-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 14:33:34 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 14:33:34 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-hard/file
FS                  : 150.0 TiB   Used FS: 50.0%   Inodes: 858.3 Mi   Used Inodes: 42.6%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 533.56 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 9 max: 298372 -- min data: 0.0 GiB mean data: 0.5 GiB time: 300.1s
WARNING: Expected aggregate file size       = 586659840000000
WARNING: Stat() of aggregate file size      = 17504286978048
WARNING: Using actual aggregate bytes moved = 17504286978048
WARNING: Maybe caused by deadlineForStonewalling
write     20210      450823     789.27      45.91      45.91      0.010618   825.97     0.003273   825.99     0   
mdtest-easy-delete
-- started at 10/18/2020 15:11:33 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.18-14.18.58/mdtest-easy' '-x' './results/2020.10.18-14.18.58/mdtest-easy.stonewall' '-r' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     451129.556     451128.434     451129.188          0.260
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.933          0.933          0.933          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        235.797        235.796        235.796          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.072          1.072          1.072          0.000
-- finished at 10/18/2020 15:15:33 --

mdtest-easy-stat
-- started at 10/18/2020 15:04:16 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.18-14.18.58/mdtest-easy' '-x' './results/2020.10.18-14.18.58/mdtest-easy.stonewall' '-T' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    4314480.217    4314459.732    4314466.567          3.132
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         24.655         24.655         24.655          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 15:04:40 --

mdtest-easy-write
-- started at 10/18/2020 14:27:12 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.18-14.18.58/mdtest-easy' '-x' './results/2020.10.18-14.18.58/mdtest-easy.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 49.7%   Inodes: 858.3 Mi   Used Inodes: 30.6%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files
Continue stonewall hit min: 37181 max: 85236 avg: 78958.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     301376.122     301374.815     301375.530          0.184
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     328441.802             NA
   Tree creation             :         16.158         16.158         16.158          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        352.964        352.963        352.963          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.023             NA
   Tree creation             :          0.062          0.062          0.062          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 14:33:34 --

mdtest-hard-delete
-- started at 10/18/2020 15:16:43 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-r' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.5%   Inodes: 858.3 Mi   Used Inodes: 35.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      81631.309      81631.165      81631.259          0.027
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.313          0.313          0.313          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        279.883        279.882        279.882          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          3.192          3.192          3.192          0.000
-- finished at 10/18/2020 15:21:27 --

mdtest-hard-read
-- started at 10/18/2020 15:15:34 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-E' '-X' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.5%   Inodes: 858.3 Mi   Used Inodes: 35.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     327238.318     327237.752     327238.148          0.108
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         69.818         69.818         69.818          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 15:16:43 --

mdtest-hard-stat
-- started at 10/18/2020 15:10:48 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-T' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     506149.501     506143.927     506145.671          0.690
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         45.140         45.139         45.139          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 15:11:33 --

mdtest-hard-write
-- started at 10/18/2020 14:47:20 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.4%   Inodes: 858.3 Mi   Used Inodes: 44.5%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files
Continue stonewall hit min: 17009 max: 18307 avg: 17585.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      69443.543      69443.324      69443.426          0.031
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      73149.139             NA
   Tree creation             :        965.317        965.317        965.317          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        329.004        329.003        329.004          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.021             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 14:53:30 --

result
version         = io500-sc20
config-hash     = F6D0CA1
result-dir      = ./results/2020.10.18-14.18.58
; START 2020-10-18 14:18:58


[ior-easy-write]
t_start         = 2020-10-18 14:18:58
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
throughput-stonewall = 62319.37
score           = 75.347071
t_delta         = 493.8916
t_end           = 2020-10-18 14:27:12

[mdtest-easy-write]
t_start         = 2020-10-18 14:27:12
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.18-14.18.58/mdtest-easy -x ./results/2020.10.18-14.18.58/mdtest-easy.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 328.441802
score           = 301.375520
t_delta         = 382.4359
t_end           = 2020-10-18 14:33:34

[timestamp]
t_start         = 2020-10-18 14:33:34
t_delta         = 0.0012
t_end           = 2020-10-18 14:33:34

[ior-hard-write]
t_start         = 2020-10-18 14:33:34
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
throughput-stonewall = 998.16
score           = 19.736564
t_delta         = 826.0049
t_end           = 2020-10-18 14:47:20

[mdtest-hard-write]
t_start         = 2020-10-18 14:47:20
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 73.149139
score           = 69.443424
t_delta         = 369.8390
t_end           = 2020-10-18 14:53:30

[find]
t_start         = 2020-10-18 14:53:30
exe             = ./pfind ./datadir/2020.10.18-14.18.58 -newer ./results/2020.10.18-14.18.58/timestampfile -size 3901c -name *01* -C -q 10000
nproc           = 480
found           = 969166
total-files     = 129222913
score           = 700.954508
t_delta         = 184.3682
t_end           = 2020-10-18 14:56:34

[ior-easy-read]
t_start         = 2020-10-18 14:56:34
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -r -R -a POSIX
score           = 80.670930
t_delta         = 461.2849
t_end           = 2020-10-18 15:04:16

[mdtest-easy-stat]
t_start         = 2020-10-18 15:04:16
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.18-14.18.58/mdtest-easy -x ./results/2020.10.18-14.18.58/mdtest-easy.stonewall -T -a POSIX
score           = 4314.479508
t_delta         = 24.7125
t_end           = 2020-10-18 15:04:40

[ior-hard-read]
t_start         = 2020-10-18 15:04:40
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
score           = 44.352347
t_delta         = 367.5758
t_end           = 2020-10-18 15:10:48

[mdtest-hard-stat]
t_start         = 2020-10-18 15:10:48
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -T -a POSIX
score           = 506.145766
t_delta         = 45.1544
t_end           = 2020-10-18 15:11:33

[mdtest-easy-delete]
t_start         = 2020-10-18 15:11:33
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.18-14.18.58/mdtest-easy -x ./results/2020.10.18-14.18.58/mdtest-easy.stonewall -r -a POSIX
score           = 451.129284
t_delta         = 240.2008
t_end           = 2020-10-18 15:15:33

[mdtest-hard-read]
t_start         = 2020-10-18 15:15:33
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -E -X -a POSIX
score           = 327.238233
t_delta         = 70.0446
t_end           = 2020-10-18 15:16:43

[mdtest-hard-delete]
t_start         = 2020-10-18 15:16:43
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -r -a POSIX
score           = 81.631238
t_delta         = 283.0916
t_end           = 2020-10-18 15:21:27

[SCORE]
MD              = 374.396517
BW              = 48.027823
SCORE           = 134.094927 
hash            = DF69C0A5
; END 2020-10-18 15:21:32
result_summary
IO500 version io500-sc20
[RESULT]       ior-easy-write       75.347071 GiB/s : time 493.892 seconds
[RESULT]    mdtest-easy-write      301.375520 kIOPS : time 382.436 seconds
[RESULT]       ior-hard-write       19.736564 GiB/s : time 826.005 seconds
[RESULT]    mdtest-hard-write       69.443424 kIOPS : time 369.839 seconds
[RESULT]                 find      700.954508 kIOPS : time 184.368 seconds
[RESULT]        ior-easy-read       80.670930 GiB/s : time 461.285 seconds
[RESULT]     mdtest-easy-stat     4314.479508 kIOPS : time 24.713 seconds
[RESULT]        ior-hard-read       44.352347 GiB/s : time 367.576 seconds
[RESULT]     mdtest-hard-stat      506.145766 kIOPS : time 45.154 seconds
[RESULT]   mdtest-easy-delete      451.129284 kIOPS : time 240.201 seconds
[RESULT]     mdtest-hard-read      327.238233 kIOPS : time 70.045 seconds
[RESULT]   mdtest-hard-delete       81.631238 kIOPS : time 283.092 seconds
[SCORE] Bandwidth 48.027823 GB/s : IOPS 374.396517 kiops : TOTAL 134.094927