SharedScratch

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 192GB
Storage Interface NVMe
Network 100GbE
Software Version 3.9.0
OS Version 7.7

INFORMATION

Client Nodes 48
Client Total Procs 1,248
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 20
Data Storage Devices 80

METADATA

Easy Write 301.38 kIOP/s
Easy Stat 4,314.48 kIOP/s
Easy Delete 451.13 kIOP/s
Hard Write 69.44 kIOP/s
Hard Read 327.24 kIOP/s
Hard Stat 506.15 kIOP/s
Hard Delete 81.63 kIOP/s

Submitted Files

io500
#!/bin/bash

export I_MPI_FABRICS=shm:dapl
export I_MPI_FALLBACK=tcp
export I_MPI_DEBUG=0
export I_MPI_ROOT=/misc/local/INTEL-2018/compilers_and_libraries_2018.0.128/linux/mpi
export DAT_OVERRIDE=/misc/local/admin/cx5_dat.conf

mpirun -np 1248 -ppn 26 -f hostfile-48-mix ./io500 config-hhmi.ini

ior-easy-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 14:56:34 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 14:56:34 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy.00000000
FS                  : 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 11806.64 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 12981538652160000
WARNING: Stat() of aggregate file size      = 39949638303744
WARNING: Using actual aggregate bytes moved = 39949638303744
read      82607      2581.50    0.001488    10158080000 32768      0.006506   461.20     0.003419   461.21     0   
ior-easy-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 14:18:58 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 14:18:58 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy.00000000
FS                  : 150.0 TiB   Used FS: 25.4%   Inodes: 858.3 Mi   Used Inodes: 26.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 32 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 11806.64 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 397 max: 954 -- min data: 12.4 GiB mean data: 19.8 GiB time: 300.4s
WARNING: Expected aggregate file size       = 12981538652160000
WARNING: Stat() of aggregate file size      = 39949638303744
WARNING: Using actual aggregate bytes moved = 39949638303744
WARNING: Maybe caused by deadlineForStonewalling
write     77155      2411.39    0.001476    10158080000 32768      0.054248   493.74     0.004369   493.79     0   
ior-hard-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 15:04:40 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 15:04:40 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-hard/file
FS                  : 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 533.56 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 298372 max: 298372 -- min data: 13.1 GiB mean data: 13.1 GiB time: 367.3s
WARNING: Expected aggregate file size       = 586659840000000
WARNING: Stat() of aggregate file size      = 17504286978048
WARNING: Using actual aggregate bytes moved = 17504286978048
read      45417      1013118    367.54      45.91      45.91      0.014385   367.55     0.003304   367.56     0   
ior-hard-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Oct 18 14:33:34 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
Machine             : Linux h09u02.int.janelia.org
TestID              : 0
StartTime           : Sun Oct 18 14:33:34 2020
Path                : ./datadir/2020.10.18-14.18.58/ior-hard/file
FS                  : 150.0 TiB   Used FS: 50.0%   Inodes: 858.3 Mi   Used Inodes: 42.6%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datadir/2020.10.18-14.18.58/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 1248
clients per node    : 26
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 533.56 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 9 max: 298372 -- min data: 0.0 GiB mean data: 0.5 GiB time: 300.1s
WARNING: Expected aggregate file size       = 586659840000000
WARNING: Stat() of aggregate file size      = 17504286978048
WARNING: Using actual aggregate bytes moved = 17504286978048
WARNING: Maybe caused by deadlineForStonewalling
write     20210      450823     789.27      45.91      45.91      0.010618   825.97     0.003273   825.99     0   
mdtest-easy-delete
-- started at 10/18/2020 15:11:33 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.18-14.18.58/mdtest-easy' '-x' './results/2020.10.18-14.18.58/mdtest-easy.stonewall' '-r' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     451129.556     451128.434     451129.188          0.260
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.933          0.933          0.933          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        235.797        235.796        235.796          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.072          1.072          1.072          0.000
-- finished at 10/18/2020 15:15:33 --

mdtest-easy-stat
-- started at 10/18/2020 15:04:16 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.18-14.18.58/mdtest-easy' '-x' './results/2020.10.18-14.18.58/mdtest-easy.stonewall' '-T' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    4314480.217    4314459.732    4314466.567          3.132
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         24.655         24.655         24.655          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 15:04:40 --

mdtest-easy-write
-- started at 10/18/2020 14:27:12 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datadir/2020.10.18-14.18.58/mdtest-easy' '-x' './results/2020.10.18-14.18.58/mdtest-easy.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 49.7%   Inodes: 858.3 Mi   Used Inodes: 30.6%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files
Continue stonewall hit min: 37181 max: 85236 avg: 78958.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     301376.122     301374.815     301375.530          0.184
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     328441.802             NA
   Tree creation             :         16.158         16.158         16.158          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        352.964        352.963        352.963          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.023             NA
   Tree creation             :          0.062          0.062          0.062          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 14:33:34 --

mdtest-hard-delete
-- started at 10/18/2020 15:16:43 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-r' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.5%   Inodes: 858.3 Mi   Used Inodes: 35.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      81631.309      81631.165      81631.259          0.027
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.313          0.313          0.313          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        279.883        279.882        279.882          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          3.192          3.192          3.192          0.000
-- finished at 10/18/2020 15:21:27 --

mdtest-hard-read
-- started at 10/18/2020 15:15:34 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-E' '-X' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.5%   Inodes: 858.3 Mi   Used Inodes: 35.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     327238.318     327237.752     327238.148          0.108
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         69.818         69.818         69.818          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 15:16:43 --

mdtest-hard-stat
-- started at 10/18/2020 15:10:48 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-T' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.7%   Inodes: 858.3 Mi   Used Inodes: 47.0%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     506149.501     506143.927     506145.671          0.690
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         45.140         45.139         45.139          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 15:11:33 --

mdtest-hard-write
-- started at 10/18/2020 14:47:20 --

mdtest-3.4.0+dev was launched with 1248 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datadir/2020.10.18-14.18.58/mdtest-hard' '-x' './results/2020.10.18-14.18.58/mdtest-hard.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /misc/weka/wekaFS_1/io500-sc20/io500/datadir/2020.10.18-14.18.58
FS: 150.0 TiB   Used FS: 60.4%   Inodes: 858.3 Mi   Used Inodes: 44.5%

Nodemap: 111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 26 for each phase.
1248 tasks, 1248000000 files
Continue stonewall hit min: 17009 max: 18307 avg: 17585.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      69443.543      69443.324      69443.426          0.031
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      73149.139             NA
   Tree creation             :        965.317        965.317        965.317          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        329.004        329.003        329.004          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.021             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/18/2020 14:53:30 --

result
version         = io500-sc20
config-hash     = F6D0CA1
result-dir      = ./results/2020.10.18-14.18.58
; START 2020-10-18 14:18:58


[ior-easy-write]
t_start         = 2020-10-18 14:18:58
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
throughput-stonewall = 62319.37
score           = 75.347071
t_delta         = 493.8916
t_end           = 2020-10-18 14:27:12

[mdtest-easy-write]
t_start         = 2020-10-18 14:27:12
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.18-14.18.58/mdtest-easy -x ./results/2020.10.18-14.18.58/mdtest-easy.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 328.441802
score           = 301.375520
t_delta         = 382.4359
t_end           = 2020-10-18 14:33:34

[timestamp]
t_start         = 2020-10-18 14:33:34
t_delta         = 0.0012
t_end           = 2020-10-18 14:33:34

[ior-hard-write]
t_start         = 2020-10-18 14:33:34
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
throughput-stonewall = 998.16
score           = 19.736564
t_delta         = 826.0049
t_end           = 2020-10-18 14:47:20

[mdtest-hard-write]
t_start         = 2020-10-18 14:47:20
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 73.149139
score           = 69.443424
t_delta         = 369.8390
t_end           = 2020-10-18 14:53:30

[find]
t_start         = 2020-10-18 14:53:30
exe             = ./pfind ./datadir/2020.10.18-14.18.58 -newer ./results/2020.10.18-14.18.58/timestampfile -size 3901c -name *01* -C -q 10000
nproc           = 480
found           = 969166
total-files     = 129222913
score           = 700.954508
t_delta         = 184.3682
t_end           = 2020-10-18 14:56:34

[ior-easy-read]
t_start         = 2020-10-18 14:56:34
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datadir/2020.10.18-14.18.58/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-easy.stonewall -t 32m -b 9920000m -F -r -R -a POSIX
score           = 80.670930
t_delta         = 461.2849
t_end           = 2020-10-18 15:04:16

[mdtest-easy-stat]
t_start         = 2020-10-18 15:04:16
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.18-14.18.58/mdtest-easy -x ./results/2020.10.18-14.18.58/mdtest-easy.stonewall -T -a POSIX
score           = 4314.479508
t_delta         = 24.7125
t_end           = 2020-10-18 15:04:40

[ior-hard-read]
t_start         = 2020-10-18 15:04:40
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datadir/2020.10.18-14.18.58/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.18-14.18.58/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
score           = 44.352347
t_delta         = 367.5758
t_end           = 2020-10-18 15:10:48

[mdtest-hard-stat]
t_start         = 2020-10-18 15:10:48
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -T -a POSIX
score           = 506.145766
t_delta         = 45.1544
t_end           = 2020-10-18 15:11:33

[mdtest-easy-delete]
t_start         = 2020-10-18 15:11:33
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datadir/2020.10.18-14.18.58/mdtest-easy -x ./results/2020.10.18-14.18.58/mdtest-easy.stonewall -r -a POSIX
score           = 451.129284
t_delta         = 240.2008
t_end           = 2020-10-18 15:15:33

[mdtest-hard-read]
t_start         = 2020-10-18 15:15:33
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -E -X -a POSIX
score           = 327.238233
t_delta         = 70.0446
t_end           = 2020-10-18 15:16:43

[mdtest-hard-delete]
t_start         = 2020-10-18 15:16:43
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datadir/2020.10.18-14.18.58/mdtest-hard -x ./results/2020.10.18-14.18.58/mdtest-hard.stonewall -r -a POSIX
score           = 81.631238
t_delta         = 283.0916
t_end           = 2020-10-18 15:21:27

[SCORE]
MD              = 374.396517
BW              = 48.027823
SCORE           = 134.094927 
hash            = DF69C0A5
; END 2020-10-18 15:21:32
result_summary
IO500 version io500-sc20
[RESULT]       ior-easy-write       75.347071 GiB/s : time 493.892 seconds
[RESULT]    mdtest-easy-write      301.375520 kIOPS : time 382.436 seconds
[RESULT]       ior-hard-write       19.736564 GiB/s : time 826.005 seconds
[RESULT]    mdtest-hard-write       69.443424 kIOPS : time 369.839 seconds
[RESULT]                 find      700.954508 kIOPS : time 184.368 seconds
[RESULT]        ior-easy-read       80.670930 GiB/s : time 461.285 seconds
[RESULT]     mdtest-easy-stat     4314.479508 kIOPS : time 24.713 seconds
[RESULT]        ior-hard-read       44.352347 GiB/s : time 367.576 seconds
[RESULT]     mdtest-hard-stat      506.145766 kIOPS : time 45.154 seconds
[RESULT]   mdtest-easy-delete      451.129284 kIOPS : time 240.201 seconds
[RESULT]     mdtest-hard-read      327.238233 kIOPS : time 70.045 seconds
[RESULT]   mdtest-hard-delete       81.631238 kIOPS : time 283.092 seconds
[SCORE] Bandwidth 48.027823 GB/s : IOPS 374.396517 kiops : TOTAL 134.094927