JASMIN 4

Institution The Science and Technology Facilities Council
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8.2003
Client Kernel Version 3.10.0-1127.10.1.el7.x86_64

DATA SERVER

Storage Type each host: 1x SASSATA SDD (metadata); 44x SAS-NL (data); 1x SAS-SSD (data; 47 hosts only)
Volatile Memory 64GiB
Storage Interface SAS
Network Ethernet
Software Version Quobyte 2.26
OS Version RHEL 7.5

INFORMATION

Client Nodes 10
Client Total Procs 240
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 87
Data Storage Devices 4,546

METADATA

Easy Write 4.46 kIOP/s
Easy Stat 38.38 kIOP/s
Easy Delete 9.59 kIOP/s
Hard Write 2.22 kIOP/s
Hard Read 4.87 kIOP/s
Hard Stat 8.35 kIOP/s
Hard Delete 5.76 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 20:11:21 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 20:11:21 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy
FS                  : 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 68.66 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 75489418936320.
WARNING: Stat() of aggregate file size      = 28831981240320.
WARNING: Using actual aggregate bytes moved = 28831981240320.
read      47519      5954       0.015395    307167232  8192       1.38       577.24     0.015884   578.64     0   
Max Read:  47518.99 MiB/sec (49827.27 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        47518.99   47518.99   47518.99       0.00    5939.87    5939.87    5939.87       0.00  578.63855         NA            NA     0    240  24    1   1     1        1         0    0      1 314539245568  8388608 27496320.0 MPIIO      0
Finished            : Sat Oct 17 20:21:00 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 19:47:41 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 19:47:41 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy
FS                  : 931.3 TiB   Used FS: 3.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 8 MiB
blocksize           : 292.94 GiB
aggregate filesize  : 68.66 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 11521 max: 14321 -- min data: 90.0 GiB mean data: 102.4 GiB time: 300.5s
WARNING: Expected aggregate file size       = 75489418936320.
WARNING: Stat() of aggregate file size      = 28831981240320.
WARNING: Using actual aggregate bytes moved = 28831981240320.
WARNING: maybe caused by deadlineForStonewalling
write     79798      9992       0.008518    307167232  8192       0.587833   343.98     0.007568   344.57     0   
Max Write: 79797.78 MiB/sec (83674.04 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       79797.78   79797.78   79797.78       0.00    9974.72    9974.72    9974.72       0.00  344.57499     300.48      83778.74     0    240  24    1   1     1        1         0    0      1 314539245568  8388608 27496320.0 MPIIO      0
Finished            : Sat Oct 17 19:53:27 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 20:21:47 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 20:21:47 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard
FS                  : 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 102.61 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 83634 max: 83634 -- min data: 3.7 GiB mean data: 3.7 GiB time: 372.9s
WARNING: Expected aggregate file size       = 112819200000000.
WARNING: Stat() of aggregate file size      = 943552097280.
WARNING: Using actual aggregate bytes moved = 943552097280.
read      2301.60    53809      373.01      45.91      45.91      17.90      373.02     0.044347   390.96     0   
Max Read:  2301.60 MiB/sec (2413.41 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         2301.60    2301.60    2301.60       0.00   51340.31   51340.31   51340.31       0.00  390.96294     372.87       2413.26     0    240  24    1   0     1        1         0    0 10000000    47008    47008  899841.4 MPIIO      0
Finished            : Sat Oct 17 20:28:18 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Oct 17 20:00:16 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
Machine             : Linux host639.jc.rl.ac.uk
TestID              : 0
StartTime           : Sat Oct 17 20:00:16 2020
Path                : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard
FS                  : 931.3 TiB   Used FS: 5.9%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 240
clients per node    : 24
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 102.61 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 81961 max: 83634 -- min data: 3.6 GiB mean data: 3.6 GiB time: 300.0s
WARNING: Expected aggregate file size       = 112819200000000.
WARNING: Stat() of aggregate file size      = 943552097280.
WARNING: Using actual aggregate bytes moved = 943552097280.
WARNING: maybe caused by deadlineForStonewalling
write     2920.74    65813      301.23      45.91      45.91      3.05       304.99     0.051852   308.09     0   
Max Write: 2920.74 MiB/sec (3062.62 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2920.74    2920.74    2920.74       0.00   65151.08   65151.08   65151.08       0.00  308.08636     300.02       2962.38     0    240  24    1   0     1        1         0    0 10000000    47008    47008  899841.4 MPIIO      0
Finished            : Sat Oct 17 20:05:24 2020
mdtest-easy-delete
-- started at 10/17/2020 20:29:45 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       9594.347       9593.842       9594.166          0.053
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.087          2.087          2.087          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        182.317        182.307        182.311          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.479          0.479          0.479          0.000
-- finished at 10/17/2020 20:32:49 --

mdtest-easy-stat
-- started at 10/17/2020 20:21:01 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      38384.504      38361.162      38378.763          2.048
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         45.596         45.568         45.575          0.002
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:21:46 --

mdtest-easy-write
-- started at 10/17/2020 19:53:27 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 5.9%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files
Continue stonewall hit min: 5180 max: 7288 avg: 5913.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       4461.971       4461.965       4461.968          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       4730.757             NA
   Tree creation             :         20.355         20.355         20.355          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        392.007        392.006        392.006          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.977             NA
   Tree creation             :          0.049          0.049          0.049          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:00:15 --

mdtest-hard-delete
-- started at 10/17/2020 20:35:16 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       5755.238       5754.228       5755.053          0.132
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.505          0.505          0.505          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        123.749        123.727        123.731          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.980          1.980          1.980          0.000
-- finished at 10/17/2020 20:37:23 --

mdtest-hard-read
-- started at 10/17/2020 20:32:49 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :       4867.289       4866.725       4867.189          0.065
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        146.316        146.299        146.302          0.002
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:35:16 --

mdtest-hard-stat
-- started at 10/17/2020 20:28:19 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 6.0%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :       8355.127       8353.843       8354.593          0.120
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         85.240         85.227         85.232          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:29:44 --

mdtest-hard-write
-- started at 10/17/2020 20:05:25 --

mdtest-3.3.0+dev was launched with 240 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '35000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard' '-x' '/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app
FS: 931.3 TiB   Used FS: 5.9%   Inodes: 8796093022208.0 Mi   Used Inodes: 0.0%

Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
240 tasks, 8400000 files
Continue stonewall hit min: 2763 max: 2967 avg: 2854.7 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2220.931       2220.927       2220.928          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2283.033             NA
   Tree creation             :        589.992        589.992        589.992          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        320.623        320.622        320.623          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.095             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/17/2020 20:11:06 --

result
version         = io500-isc20_v4-4-g5f3675422b54
config-hash     = 93A3A972
result-dir      = /tmp/io500-results/all_240_m35k/2020.10.17-19.02.11-app
; START 2020-10-17 19:47:41


[ior-easy-write]
t_start         = 2020-10-17 19:47:41
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -w -D 300 -O stoneWallingWearOut=1 -a MPIIO
throughput-stonewall = 17123.00
score           = 77.927521
t_delta         = 346.0424
t_end           = 2020-10-17 19:53:27

[mdtest-easy-write]
t_start         = 2020-10-17 19:53:27
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 4.730757
score           = 4.461971
t_delta         = 407.6110
t_end           = 2020-10-17 20:00:15

[timestamp]
t_start         = 2020-10-17 20:00:15
t_delta         = 1.1151
t_end           = 2020-10-17 20:00:16

[ior-hard-write]
t_start         = 2020-10-17 20:00:16
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a MPIIO
throughput-stonewall = 676.12
score           = 2.852289
t_delta         = 308.6370
t_end           = 2020-10-17 20:05:24

[mdtest-hard-write]
t_start         = 2020-10-17 20:05:25
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 2.283033
score           = 2.220931
t_delta         = 340.9704
t_end           = 2020-10-17 20:11:06

[find]
t_start         = 2020-10-17 20:11:06
exe             = ./pfind /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app -newer /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 17596
total-files     = 2461446
score           = 163.496717
t_delta         = 15.1029
t_end           = 2020-10-17 20:11:21

[ior-easy-read]
t_start         = 2020-10-17 20:11:21
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-easy/stonewall -t 8m -b 299968m -F -r -R -a MPIIO
score           = 46.405265
t_delta         = 579.5138
t_end           = 2020-10-17 20:21:00

[mdtest-easy-stat]
t_start         = 2020-10-17 20:21:00
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall -T -a POSIX
score           = 38.379075
t_delta         = 46.1526
t_end           = 2020-10-17 20:21:46

[ior-hard-read]
t_start         = 2020-10-17 20:21:46
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/file -O stoneWallingStatusFile=/var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a MPIIO
score           = 2.247659
t_delta         = 391.8735
t_end           = 2020-10-17 20:28:18

[mdtest-hard-stat]
t_start         = 2020-10-17 20:28:18
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -T -a POSIX
score           = 8.354663
t_delta         = 85.9155
t_end           = 2020-10-17 20:29:44

[mdtest-easy-delete]
t_start         = 2020-10-17 20:29:44
exe             = ./mdtest -n 35000 -u -L -F -P -N 1 -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-easy-stonewall -r -a POSIX
score           = 9.594175
t_delta         = 184.7221
t_end           = 2020-10-17 20:32:49

[mdtest-hard-read]
t_start         = 2020-10-17 20:32:49
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 4.867108
t_delta         = 146.8201
t_end           = 2020-10-17 20:35:16

[mdtest-hard-delete]
t_start         = 2020-10-17 20:35:16
exe             = ./mdtest -n 35000 -t -w 3901 -e 3901 -P -N 1 -F -d /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard -x /var/tmp/gws_quobyte-replicated/r2020/datafiles_all/2020.10.17-19.02.11-app/mdtest-hard-stonewall -r -a POSIX
score           = 5.755092
t_delta         = 127.6174
t_end           = 2020-10-17 20:37:23

[SCORE]
MD              = 10.425950
BW              = 12.339442
SCORE           = 11.342416 
hash            = 54C98E5C
; END 2020-10-17 20:37:37
result_summary
IO500 version io500-isc20_v4-4-g5f3675422b54
[RESULT]       ior-easy-write       77.927521 GiB/s  : time 346.042 seconds
[RESULT]    mdtest-easy-write        4.461971 kIOPS : time 407.611 seconds
[RESULT]       ior-hard-write        2.852289 GiB/s  : time 308.637 seconds
[RESULT]    mdtest-hard-write        2.220931 kIOPS : time 340.970 seconds
[RESULT]                 find      163.496717 kIOPS : time 15.103 seconds
[RESULT]        ior-easy-read       46.405265 GiB/s  : time 579.514 seconds
[RESULT]     mdtest-easy-stat       38.379075 kIOPS : time 46.153 seconds
[RESULT]        ior-hard-read        2.247659 GiB/s  : time 391.874 seconds
[RESULT]     mdtest-hard-stat        8.354663 kIOPS : time 85.916 seconds
[RESULT]   mdtest-easy-delete        9.594175 kIOPS : time 184.722 seconds
[RESULT]     mdtest-hard-read        4.867108 kIOPS : time 146.820 seconds
[RESULT]   mdtest-hard-delete        5.755092 kIOPS : time 127.617 seconds
[SCORE] Bandwidth 12.339442 GB/s : IOPS 10.425950 kiops : TOTAL 11.342416