JASMIN 4

Institution The Science and Technology Facilities Council
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8.2003
Client Kernel Version 3.10.0-1127.10.1.el7.x86_64

DATA SERVER

Storage Type each host: 1x SASSATA SDD (metadata); 44x SAS-NL (data); 1x SAS-SSD (data; 47 hosts only)
Volatile Memory 64GiB
Storage Interface SAS
Network Ethernet
Software Version Quobyte 2.26
OS Version RHEL 7.5

INFORMATION

Client Nodes 48
Client Total Procs 336

METADATA

Easy Write 4.64 kIOP/s
Easy Stat 56.54 kIOP/s
Easy Delete 10.70 kIOP/s
Hard Write 2.87 kIOP/s
Hard Read 7.84 kIOP/s
Hard Stat 47.41 kIOP/s
Hard Delete 8.19 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:50:11 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:50:11 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy
FS                  : 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 96.12 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 105685186510848.
WARNING: Stat() of aggregate file size      = 65934861533184.
WARNING: Using actual aggregate bytes moved = 65934861533184.
read      117537     14778      0.014185    307167232  8192       3.10       531.88     0.005891   534.99     0   
Max Read:  117536.51 MiB/sec (123245.96 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       117536.51  117536.51  117536.51       0.00   14692.06   14692.06   14692.06       0.00  534.98598         NA            NA     0    336   7    1   1     1        1         0    0      1 314539245568  8388608 62880384.0 MPIIO      0
Finished            : Sat Oct 17 17:59:07 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:24:23 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:24:23 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy
FS                  : 931.3 TiB   Used FS: 0.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 96.12 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 8193 max: 23393 -- min data: 64.0 GiB mean data: 131.2 GiB time: 305.4s
WARNING: Expected aggregate file size       = 105685186510848.
WARNING: Stat() of aggregate file size      = 65934861533184.
WARNING: Using actual aggregate bytes moved = 65934861533184.
WARNING: maybe caused by deadlineForStonewalling
write     123857     15506      0.011020    307167232  8192       0.788024   506.90     0.001423   507.68     0   
Max Write: 123857.09 MiB/sec (129873.57 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      123857.09  123857.09  123857.09       0.00   15482.14   15482.14   15482.14       0.00  507.68499     305.36     147809.54     0    336   7    1   1     1        1         0    0      1 314539245568  8388608 62880384.0 MPIIO      0
Finished            : Sat Oct 17 17:32:52 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:59:36 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:59:36 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard
FS                  : 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 143.65 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 66603 max: 66603 -- min data: 2.9 GiB mean data: 2.9 GiB time: 261.1s
WARNING: Expected aggregate file size       = 157946880000000.
WARNING: Stat() of aggregate file size      = 1051973604864.
WARNING: Using actual aggregate bytes moved = 1051973604864.
read      3622       85675      261.20      45.91      45.91      15.74      261.20     0.016833   276.96     0   
Max Read:  3622.33 MiB/sec (3798.29 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         3622.33    3622.33    3622.33       0.00   80800.99   80800.99   80800.99       0.00  276.95958     261.10       3842.31     0    336   7    1   0     1        1         0    0 10000000    47008    47008 1003240.2 MPIIO      0
Finished            : Sat Oct 17 18:04:14 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:38:39 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:38:39 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard
FS                  : 931.3 TiB   Used FS: 6.4%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 143.65 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 59725 max: 66603 -- min data: 2.6 GiB mean data: 2.9 GiB time: 300.0s
WARNING: Expected aggregate file size       = 157946880000000.
WARNING: Stat() of aggregate file size      = 1051973604864.
WARNING: Using actual aggregate bytes moved = 1051973604864.
WARNING: maybe caused by deadlineForStonewalling
write     3176.99    71495      304.21      45.91      45.91      2.72       313.01     0.055420   315.78     0   
Max Write: 3176.99 MiB/sec (3331.31 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        3176.99    3176.99    3176.99       0.00   70866.92   70866.92   70866.92       0.00  315.78356     300.05       3292.72     0    336   7    1   0     1        1         0    0 10000000    47008    47008 1003240.2 MPIIO      0
Finished            : Sat Oct 17 17:43:55 2020
mdtest-easy-delete
-- started at 10/17/2020 18:04:36 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      10704.268      10703.998      10704.130          0.051
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.398          2.398          2.398          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        149.888        149.884        149.886          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.417          0.417          0.417          0.000
-- finished at 10/17/2020 18:07:07 --

mdtest-easy-stat
-- started at 10/17/2020 17:59:08 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      56548.463      56539.662      56544.368          1.425
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         28.377         28.372         28.374          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 17:59:36 --

mdtest-easy-write
-- started at 10/17/2020 17:32:52 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.4%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files
Continue stonewall hit min: 3872 max: 4775 avg: 4227.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       4637.821       4637.814       4637.818          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       4733.727             NA
   Tree creation             :         29.993         29.993         29.993          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        345.939        345.938        345.939          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.033             NA
   Tree creation             :          0.033          0.033          0.033          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 17:38:39 --

mdtest-hard-delete
-- started at 10/17/2020 18:09:18 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       8192.665       8192.403       8192.571          0.045
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.180          0.180          0.180          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        124.845        124.841        124.843          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          5.556          5.556          5.556          0.000
-- finished at 10/17/2020 18:11:28 --

mdtest-hard-read
-- started at 10/17/2020 18:07:07 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :       7840.237       7839.997       7840.098          0.047
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        130.457        130.453        130.456          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 18:09:18 --

mdtest-hard-stat
-- started at 10/17/2020 18:04:14 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      47417.013      47409.555      47413.146          1.474
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         21.573         21.570         21.572          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 18:04:35 --

mdtest-hard-write
-- started at 10/17/2020 17:43:55 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files
Continue stonewall hit min: 2322 max: 3044 avg: 2591.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2870.380       2870.371       2870.377          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2901.057             NA
   Tree creation             :        520.968        520.968        520.968          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        356.325        356.324        356.324          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.105             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 17:49:53 --

result
version         = io500-isc20_v4-4-g5f3675422b54
config-hash     = 93A3E172
result-dir      = /tmp/io500-results/all_336_m35k/2020.10.17-17.24.08-app
; START 2020-10-17 17:24:23


[ior-easy-write]
t_start         = 2020-10-17 17:24:23
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
throughput-stonewall = 29171.42
score           = 120.954186
t_delta         = 509.5725
t_end           = 2020-10-17 17:32:52

[mdtest-easy-write]
t_start         = 2020-10-17 17:32:52
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 4.733727
score           = 4.637818
t_delta         = 346.8589
t_end           = 2020-10-17 17:38:39

[timestamp]
t_start         = 2020-10-17 17:38:39
t_delta         = 0.0048
t_end           = 2020-10-17 17:38:39

[ior-hard-write]
t_start         = 2020-10-17 17:38:39
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
throughput-stonewall = 1026.58
score           = 3.102526
t_delta         = 316.2047
t_end           = 2020-10-17 17:43:55

[mdtest-hard-write]
t_start         = 2020-10-17 17:43:55
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 2.901057
score           = 2.870377
t_delta         = 357.1602
t_end           = 2020-10-17 17:49:53

[find]
t_start         = 2020-10-17 17:49:53
exe             = ./pfind /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app -newer /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 29112
total-files     = 2627526
score           = 142.668998
t_delta         = 18.4781
t_end           = 2020-10-17 17:50:11

[ior-easy-read]
t_start         = 2020-10-17 17:50:11
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
score           = 114.781746
t_delta         = 536.4693
t_end           = 2020-10-17 17:59:07

[mdtest-easy-stat]
t_start         = 2020-10-17 17:59:07
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall -T -a POSIX
score           = 56.543761
t_delta         = 28.5461
t_end           = 2020-10-17 17:59:36

[ior-hard-read]
t_start         = 2020-10-17 17:59:36
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
score           = 3.537436
t_delta         = 277.7112
t_end           = 2020-10-17 18:04:14

[mdtest-hard-stat]
t_start         = 2020-10-17 18:04:14
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -T -a POSIX
score           = 47.413782
t_delta         = 21.7433
t_end           = 2020-10-17 18:04:35

[mdtest-easy-delete]
t_start         = 2020-10-17 18:04:35
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall -r -a POSIX
score           = 10.704086
t_delta         = 151.4542
t_end           = 2020-10-17 18:07:07

[mdtest-hard-read]
t_start         = 2020-10-17 18:07:07
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 7.840101
t_delta         = 130.5998
t_end           = 2020-10-17 18:09:18

[mdtest-hard-delete]
t_start         = 2020-10-17 18:09:18
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -r -a POSIX
score           = 8.192642
t_delta         = 130.5821
t_end           = 2020-10-17 18:11:28

[SCORE]
MD              = 15.596274
BW              = 19.757157
SCORE           = 17.553861 
hash            = D57CB431
; END 2020-10-17 18:11:39
result_summary
IO500 version io500-isc20_v4-4-g5f3675422b54
[RESULT]       ior-easy-write      120.954186 GiB/s  : time 509.573 seconds
[RESULT]    mdtest-easy-write        4.637818 kIOPS : time 346.859 seconds
[RESULT]       ior-hard-write        3.102526 GiB/s  : time 316.205 seconds
[RESULT]    mdtest-hard-write        2.870377 kIOPS : time 357.160 seconds
[RESULT]                 find      142.668998 kIOPS : time 18.478 seconds
[RESULT]        ior-easy-read      114.781746 GiB/s  : time 536.469 seconds
[RESULT]     mdtest-easy-stat       56.543761 kIOPS : time 28.546 seconds
[RESULT]        ior-hard-read        3.537436 GiB/s  : time 277.711 seconds
[RESULT]     mdtest-hard-stat       47.413782 kIOPS : time 21.743 seconds
[RESULT]   mdtest-easy-delete       10.704086 kIOPS : time 151.454 seconds
[RESULT]     mdtest-hard-read        7.840101 kIOPS : time 130.600 seconds
[RESULT]   mdtest-hard-delete        8.192642 kIOPS : time 130.582 seconds
[SCORE] Bandwidth 19.757157 GB/s : IOPS 15.596274 kiops : TOTAL 17.553861