JASMIN 4

Institution The Science and Technology Facilities Council
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8.2003
Client Kernel Version 3.10.0-1127.10.1.el7.x86_64

DATA SERVER

Storage Type each host: 1x SASSATA SDD (metadata); 44x SAS-NL (data); 1x SAS-SSD (data; 47 hosts only)
Volatile Memory 64GiB
Storage Interface SAS
Network Ethernet
Software Version Quobyte 2.26
OS Version RHEL 7.5

INFORMATION

Client Nodes 10
Client Total Procs 240

METADATA

Easy Write 4.46 kIOP/s
Easy Stat 38.38 kIOP/s
Easy Delete 9.59 kIOP/s
Hard Write 2.22 kIOP/s
Hard Read 4.87 kIOP/s
Hard Stat 8.35 kIOP/s
Hard Delete 5.76 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 20:11:21 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 20:11:21 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy
FS                  : 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 68.66 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 75489418936320.
WARNING: Stat() of aggregate file size      = 28831981240320.
WARNING: Using actual aggregate bytes moved = 28831981240320.
read      47519      5954       0.015395    307167232  8192       1.38       577.24     0.015884   578.64     0   
Max Read:  47518.99 MiB/sec (49827.27 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        47518.99   47518.99   47518.99       0.00    5939.87    5939.87    5939.87       0.00  578.63855         NA            NA     0    240  24    1   1     1        1         0    0      1 314539245568  8388608 27496320.0 MPIIO      0
Finished            : Sat Oct 17 20:21:00 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 19:47:41 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 19:47:41 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy
FS                  : 931.3 TiB   Used FS: 3.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 68.66 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 11521 max: 14321 -- min data: 90.0 GiB mean data: 102.4 GiB time: 300.5s
WARNING: Expected aggregate file size       = 75489418936320.
WARNING: Stat() of aggregate file size      = 28831981240320.
WARNING: Using actual aggregate bytes moved = 28831981240320.
WARNING: maybe caused by deadlineForStonewalling
write     79798      9992       0.008518    307167232  8192       0.587833   343.98     0.007568   344.57     0   
Max Write: 79797.78 MiB/sec (83674.04 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       79797.78   79797.78   79797.78       0.00    9974.72    9974.72    9974.72       0.00  344.57499     300.48      83778.74     0    240  24    1   1     1        1         0    0      1 314539245568  8388608 27496320.0 MPIIO      0
Finished            : Sat Oct 17 19:53:27 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 20:21:47 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 20:21:47 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard
FS                  : 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 102.61 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 83634 max: 83634 -- min data: 3.7 GiB mean data: 3.7 GiB time: 372.9s
WARNING: Expected aggregate file size       = 112819200000000.
WARNING: Stat() of aggregate file size      = 943552097280.
WARNING: Using actual aggregate bytes moved = 943552097280.
read      2301.60    53809      373.01      45.91      45.91      17.90      373.02     0.044347   390.96     0   
Max Read:  2301.60 MiB/sec (2413.41 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         2301.60    2301.60    2301.60       0.00   51340.31   51340.31   51340.31       0.00  390.96294     372.87       2413.26     0    240  24    1   0     1        1         0    0 10000000    47008    47008  899841.4 MPIIO      0
Finished            : Sat Oct 17 20:28:18 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 20:00:16 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 20:00:16 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard
FS                  : 931.3 TiB   Used FS: 5.9%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 102.61 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 81961 max: 83634 -- min data: 3.6 GiB mean data: 3.6 GiB time: 300.0s
WARNING: Expected aggregate file size       = 112819200000000.
WARNING: Stat() of aggregate file size      = 943552097280.
WARNING: Using actual aggregate bytes moved = 943552097280.
WARNING: maybe caused by deadlineForStonewalling
write     2920.74    65813      301.23      45.91      45.91      3.05       304.99     0.051852   308.09     0   
Max Write: 2920.74 MiB/sec (3062.62 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2920.74    2920.74    2920.74       0.00   65151.08   65151.08   65151.08       0.00  308.08636     300.02       2962.38     0    240  24    1   0     1        1         0    0 10000000    47008    47008  899841.4 MPIIO      0
Finished            : Sat Oct 17 20:05:24 2020
mdtest-easy-delete
-- started at 10/17/2020 20:29:45 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       9594.347       9593.842       9594.166          0.053
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.087          2.087          2.087          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        182.317        182.307        182.311          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.479          0.479          0.479          0.000
-- finished at 10/17/2020 20:32:49 --

mdtest-easy-stat
-- started at 10/17/2020 20:21:01 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      38384.504      38361.162      38378.763          2.048
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         45.596         45.568         45.575          0.002
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:21:46 --

mdtest-easy-write
-- started at 10/17/2020 19:53:27 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 5.9%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files
Continue stonewall hit min: 5180 max: 7288 avg: 5913.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       4461.971       4461.965       4461.968          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       4730.757             NA
   Tree creation             :         20.355         20.355         20.355          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        392.007        392.006        392.006          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.977             NA
   Tree creation             :          0.049          0.049          0.049          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:00:15 --

mdtest-hard-delete
-- started at 10/17/2020 20:35:16 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       5755.238       5754.228       5755.053          0.132
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.505          0.505          0.505          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        123.749        123.727        123.731          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.980          1.980          1.980          0.000
-- finished at 10/17/2020 20:37:23 --

mdtest-hard-read
-- started at 10/17/2020 20:32:49 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :       4867.289       4866.725       4867.189          0.065
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        146.316        146.299        146.302          0.002
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:35:16 --

mdtest-hard-stat
-- started at 10/17/2020 20:28:19 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :       8355.127       8353.843       8354.593          0.120
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         85.240         85.227         85.232          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:29:44 --

mdtest-hard-write
-- started at 10/17/2020 20:05:25 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 5.9%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files
Continue stonewall hit min: 2763 max: 2967 avg: 2854.7 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2220.931       2220.927       2220.928          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2283.033             NA
   Tree creation             :        589.992        589.992        589.992          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        320.623        320.622        320.623          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.095             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:11:06 --

result
version         = io500-isc20_v4-4-g5f3675422b54
config-hash     = 93A3A972
result-dir      = /tmp/io500-results/all_240_m35k/2020.10.17-19.02.11-app
; START 2020-10-17 19:47:41


[ior-easy-write]
t_start         = 2020-10-17 19:47:41
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
throughput-stonewall = 17123.00
score           = 77.927521
t_delta         = 346.0424
t_end           = 2020-10-17 19:53:27

[mdtest-easy-write]
t_start         = 2020-10-17 19:53:27
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 4.730757
score           = 4.461971
t_delta         = 407.6110
t_end           = 2020-10-17 20:00:15

[timestamp]
t_start         = 2020-10-17 20:00:15
t_delta         = 1.1151
t_end           = 2020-10-17 20:00:16

[ior-hard-write]
t_start         = 2020-10-17 20:00:16
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
throughput-stonewall = 676.12
score           = 2.852289
t_delta         = 308.6370
t_end           = 2020-10-17 20:05:24

[mdtest-hard-write]
t_start         = 2020-10-17 20:05:25
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 2.283033
score           = 2.220931
t_delta         = 340.9704
t_end           = 2020-10-17 20:11:06

[find]
t_start         = 2020-10-17 20:11:06
exe             = ./pfind /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app -newer /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 17596
total-files     = 2461446
score           = 163.496717
t_delta         = 15.1029
t_end           = 2020-10-17 20:11:21

[ior-easy-read]
t_start         = 2020-10-17 20:11:21
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
score           = 46.405265
t_delta         = 579.5138
t_end           = 2020-10-17 20:21:00

[mdtest-easy-stat]
t_start         = 2020-10-17 20:21:00
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall -T -a POSIX
score           = 38.379075
t_delta         = 46.1526
t_end           = 2020-10-17 20:21:46

[ior-hard-read]
t_start         = 2020-10-17 20:21:46
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
score           = 2.247659
t_delta         = 391.8735
t_end           = 2020-10-17 20:28:18

[mdtest-hard-stat]
t_start         = 2020-10-17 20:28:18
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -T -a POSIX
score           = 8.354663
t_delta         = 85.9155
t_end           = 2020-10-17 20:29:44

[mdtest-easy-delete]
t_start         = 2020-10-17 20:29:44
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall -r -a POSIX
score           = 9.594175
t_delta         = 184.7221
t_end           = 2020-10-17 20:32:49

[mdtest-hard-read]
t_start         = 2020-10-17 20:32:49
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 4.867108
t_delta         = 146.8201
t_end           = 2020-10-17 20:35:16

[mdtest-hard-delete]
t_start         = 2020-10-17 20:35:16
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -r -a POSIX
score           = 5.755092
t_delta         = 127.6174
t_end           = 2020-10-17 20:37:23

[SCORE]
MD              = 10.425950
BW              = 12.339442
SCORE           = 11.342416 
hash            = 54C98E5C
; END 2020-10-17 20:37:37
result_summary
IO500 version io500-isc20_v4-4-g5f3675422b54
[RESULT]       ior-easy-write       77.927521 GiB/s  : time 346.042 seconds
[RESULT]    mdtest-easy-write        4.461971 kIOPS : time 407.611 seconds
[RESULT]       ior-hard-write        2.852289 GiB/s  : time 308.637 seconds
[RESULT]    mdtest-hard-write        2.220931 kIOPS : time 340.970 seconds
[RESULT]                 find      163.496717 kIOPS : time 15.103 seconds
[RESULT]        ior-easy-read       46.405265 GiB/s  : time 579.514 seconds
[RESULT]     mdtest-easy-stat       38.379075 kIOPS : time 46.153 seconds
[RESULT]        ior-hard-read        2.247659 GiB/s  : time 391.874 seconds
[RESULT]     mdtest-hard-stat        8.354663 kIOPS : time 85.916 seconds
[RESULT]   mdtest-easy-delete        9.594175 kIOPS : time 184.722 seconds
[RESULT]     mdtest-hard-read        4.867108 kIOPS : time 146.820 seconds
[RESULT]   mdtest-hard-delete        5.755092 kIOPS : time 127.617 seconds
[SCORE] Bandwidth 12.339442 GB/s : IOPS 10.425950 kiops : TOTAL 11.342416