JASMIN 4

Institution The Science and Technology Facilities Council
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8.2003
Client Kernel Version 3.10.0-1127.10.1.el7.x86_64

DATA SERVER

Storage Type each host: 1x SASSATA SDD (metadata); 44x SAS-NL (data); 1x SAS-SSD (data; 47 hosts only)
Volatile Memory 64GiB
Storage Interface SAS
Network Ethernet
Software Version Quobyte 2.26
OS Version RHEL 7.5

INFORMATION

Client Nodes 48
Client Total Procs 336
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 87
Data Storage Devices 4,546

METADATA

Easy Write 4.64 kIOP/s
Easy Stat 56.54 kIOP/s
Easy Delete 10.70 kIOP/s
Hard Write 2.87 kIOP/s
Hard Read 7.84 kIOP/s
Hard Stat 47.41 kIOP/s
Hard Delete 8.19 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:50:11 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:50:11 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy
FS                  : 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 96.12 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 105685186510848.
WARNING: Stat() of aggregate file size      = 65934861533184.
WARNING: Using actual aggregate bytes moved = 65934861533184.
read      117537     14778      0.014185    307167232  8192       3.10       531.88     0.005891   534.99     0   
Max Read:  117536.51 MiB/sec (123245.96 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       117536.51  117536.51  117536.51       0.00   14692.06   14692.06   14692.06       0.00  534.98598         NA            NA     0    336   7    1   1     1        1         0    0      1 314539245568  8388608 62880384.0 MPIIO      0
Finished            : Sat Oct 17 17:59:07 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:24:23 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:24:23 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy
FS                  : 931.3 TiB   Used FS: 0.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 96.12 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 8193 max: 23393 -- min data: 64.0 GiB mean data: 131.2 GiB time: 305.4s
WARNING: Expected aggregate file size       = 105685186510848.
WARNING: Stat() of aggregate file size      = 65934861533184.
WARNING: Using actual aggregate bytes moved = 65934861533184.
WARNING: maybe caused by deadlineForStonewalling
write     123857     15506      0.011020    307167232  8192       0.788024   506.90     0.001423   507.68     0   
Max Write: 123857.09 MiB/sec (129873.57 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      123857.09  123857.09  123857.09       0.00   15482.14   15482.14   15482.14       0.00  507.68499     305.36     147809.54     0    336   7    1   1     1        1         0    0      1 314539245568  8388608 62880384.0 MPIIO      0
Finished            : Sat Oct 17 17:32:52 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:59:36 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:59:36 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard
FS                  : 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 143.65 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 66603 max: 66603 -- min data: 2.9 GiB mean data: 2.9 GiB time: 261.1s
WARNING: Expected aggregate file size       = 157946880000000.
WARNING: Stat() of aggregate file size      = 1051973604864.
WARNING: Using actual aggregate bytes moved = 1051973604864.
read      3622       85675      261.20      45.91      45.91      15.74      261.20     0.016833   276.96     0   
Max Read:  3622.33 MiB/sec (3798.29 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         3622.33    3622.33    3622.33       0.00   80800.99   80800.99   80800.99       0.00  276.95958     261.10       3842.31     0    336   7    1   0     1        1         0    0 10000000    47008    47008 1003240.2 MPIIO      0
Finished            : Sat Oct 17 18:04:14 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 17:38:39 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 17:38:39 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard
FS                  : 931.3 TiB   Used FS: 6.4%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 336
clients per node    : 7
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 143.65 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 59725 max: 66603 -- min data: 2.6 GiB mean data: 2.9 GiB time: 300.0s
WARNING: Expected aggregate file size       = 157946880000000.
WARNING: Stat() of aggregate file size      = 1051973604864.
WARNING: Using actual aggregate bytes moved = 1051973604864.
WARNING: maybe caused by deadlineForStonewalling
write     3176.99    71495      304.21      45.91      45.91      2.72       313.01     0.055420   315.78     0   
Max Write: 3176.99 MiB/sec (3331.31 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        3176.99    3176.99    3176.99       0.00   70866.92   70866.92   70866.92       0.00  315.78356     300.05       3292.72     0    336   7    1   0     1        1         0    0 10000000    47008    47008 1003240.2 MPIIO      0
Finished            : Sat Oct 17 17:43:55 2020
mdtest-easy-delete
-- started at 10/17/2020 18:04:36 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      10704.268      10703.998      10704.130          0.051
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.398          2.398          2.398          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        149.888        149.884        149.886          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.417          0.417          0.417          0.000
-- finished at 10/17/2020 18:07:07 --

mdtest-easy-stat
-- started at 10/17/2020 17:59:08 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      56548.463      56539.662      56544.368          1.425
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         28.377         28.372         28.374          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 17:59:36 --

mdtest-easy-write
-- started at 10/17/2020 17:32:52 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.4%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files
Continue stonewall hit min: 3872 max: 4775 avg: 4227.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       4637.821       4637.814       4637.818          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       4733.727             NA
   Tree creation             :         29.993         29.993         29.993          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        345.939        345.938        345.939          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.033             NA
   Tree creation             :          0.033          0.033          0.033          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 17:38:39 --

mdtest-hard-delete
-- started at 10/17/2020 18:09:18 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       8192.665       8192.403       8192.571          0.045
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.180          0.180          0.180          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        124.845        124.841        124.843          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          5.556          5.556          5.556          0.000
-- finished at 10/17/2020 18:11:28 --

mdtest-hard-read
-- started at 10/17/2020 18:07:07 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :       7840.237       7839.997       7840.098          0.047
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        130.457        130.453        130.456          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 18:09:18 --

mdtest-hard-stat
-- started at 10/17/2020 18:04:14 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      47417.013      47409.555      47413.146          1.474
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         21.573         21.570         21.572          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 18:04:35 --

mdtest-hard-write
-- started at 10/17/2020 17:43:55 --

mdtest-3.3.0+dev was launched with 336 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app
FS: 931.3 TiB   Used FS: 6.5%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
336 tasks, 11760000 files
Continue stonewall hit min: 2322 max: 3044 avg: 2591.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2870.380       2870.371       2870.377          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2901.057             NA
   Tree creation             :        520.968        520.968        520.968          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        356.325        356.324        356.324          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.105             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 17:49:53 --

result
version         = io500-isc20_v4-4-g5f3675422b54
config-hash     = 93A3E172
result-dir      = /tmp/io500-results/all_336_m35k/2020.10.17-17.24.08-app
; START 2020-10-17 17:24:23


[ior-easy-write]
t_start         = 2020-10-17 17:24:23
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
throughput-stonewall = 29171.42
score           = 120.954186
t_delta         = 509.5725
t_end           = 2020-10-17 17:32:52

[mdtest-easy-write]
t_start         = 2020-10-17 17:32:52
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 4.733727
score           = 4.637818
t_delta         = 346.8589
t_end           = 2020-10-17 17:38:39

[timestamp]
t_start         = 2020-10-17 17:38:39
t_delta         = 0.0048
t_end           = 2020-10-17 17:38:39

[ior-hard-write]
t_start         = 2020-10-17 17:38:39
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
throughput-stonewall = 1026.58
score           = 3.102526
t_delta         = 316.2047
t_end           = 2020-10-17 17:43:55

[mdtest-hard-write]
t_start         = 2020-10-17 17:43:55
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 2.901057
score           = 2.870377
t_delta         = 357.1602
t_end           = 2020-10-17 17:49:53

[find]
t_start         = 2020-10-17 17:49:53
exe             = ./pfind /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app -newer /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 29112
total-files     = 2627526
score           = 142.668998
t_delta         = 18.4781
t_end           = 2020-10-17 17:50:11

[ior-easy-read]
t_start         = 2020-10-17 17:50:11
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
score           = 114.781746
t_delta         = 536.4693
t_end           = 2020-10-17 17:59:07

[mdtest-easy-stat]
t_start         = 2020-10-17 17:59:07
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall -T -a POSIX
score           = 56.543761
t_delta         = 28.5461
t_end           = 2020-10-17 17:59:36

[ior-hard-read]
t_start         = 2020-10-17 17:59:36
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
score           = 3.537436
t_delta         = 277.7112
t_end           = 2020-10-17 18:04:14

[mdtest-hard-stat]
t_start         = 2020-10-17 18:04:14
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -T -a POSIX
score           = 47.413782
t_delta         = 21.7433
t_end           = 2020-10-17 18:04:35

[mdtest-easy-delete]
t_start         = 2020-10-17 18:04:35
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-easy-stonewall -r -a POSIX
score           = 10.704086
t_delta         = 151.4542
t_end           = 2020-10-17 18:07:07

[mdtest-hard-read]
t_start         = 2020-10-17 18:07:07
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 7.840101
t_delta         = 130.5998
t_end           = 2020-10-17 18:09:18

[mdtest-hard-delete]
t_start         = 2020-10-17 18:09:18
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-17.24.08-app/mdtest-hard-stonewall -r -a POSIX
score           = 8.192642
t_delta         = 130.5821
t_end           = 2020-10-17 18:11:28

[SCORE]
MD              = 15.596274
BW              = 19.757157
SCORE           = 17.553861 
hash            = D57CB431
; END 2020-10-17 18:11:39
result_summary
IO500 version io500-isc20_v4-4-g5f3675422b54
[RESULT]       ior-easy-write      120.954186 GiB/s  : time 509.573 seconds
[RESULT]    mdtest-easy-write        4.637818 kIOPS : time 346.859 seconds
[RESULT]       ior-hard-write        3.102526 GiB/s  : time 316.205 seconds
[RESULT]    mdtest-hard-write        2.870377 kIOPS : time 357.160 seconds
[RESULT]                 find      142.668998 kIOPS : time 18.478 seconds
[RESULT]        ior-easy-read      114.781746 GiB/s  : time 536.469 seconds
[RESULT]     mdtest-easy-stat       56.543761 kIOPS : time 28.546 seconds
[RESULT]        ior-hard-read        3.537436 GiB/s  : time 277.711 seconds
[RESULT]     mdtest-hard-stat       47.413782 kIOPS : time 21.743 seconds
[RESULT]   mdtest-easy-delete       10.704086 kIOPS : time 151.454 seconds
[RESULT]     mdtest-hard-read        7.840101 kIOPS : time 130.600 seconds
[RESULT]   mdtest-hard-delete        8.192642 kIOPS : time 130.582 seconds
[SCORE] Bandwidth 19.757157 GB/s : IOPS 15.596274 kiops : TOTAL 17.553861