Officinalis (1 Client)

Institution
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version CentOS Linux release 8.0.1905 (Core)
Client Kernel Version 4.18.0-147.6.el8.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 384GB
Storage Interface NVMe
Network 4x25GbE
Software Version
OS Version CentOS Linux release 8.0.1905 (Core)

INFORMATION

Client Nodes 1
Client Total Procs 112
Metadata Nodes 8
Metadata Storage Devices 8
Data Nodes 8
Data Storage Devices 8

METADATA

Easy Write 110.70 kIOP/s
Easy Stat 165.35 kIOP/s
Easy Delete 61.98 kIOP/s
Hard Write 16.86 kIOP/s
Hard Read 43.05 kIOP/s
Hard Stat 53.90 kIOP/s
Hard Delete 9.86 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 19:22:23 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux o01
TestID              : 0
StartTime           : Mon Jul 13 19:22:23 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy
FS                  : 430.5 TiB   Used FS: 3.7%   Inodes: 52.1 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1
tasks               : 112
clients per node    : 112
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 1059.57 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 1165009879040000.
WARNING: Stat() of aggregate file size      = 6119120437248.
WARNING: Using actual aggregate bytes moved = 6119120437248.
read      9932       4966       0.000118    10158080000 2048.00    0.064065   587.52     0.039420   587.56     0   
Max Read:  9931.93 MiB/sec (10414.38 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         9931.93    9931.93    9931.93       0.00    4965.97    4965.97    4965.97       0.00  587.56430         NA            NA     0    112 112    1   1     1        1         0    0      1 10401873920000  2097152 5835648.0 CEPHFS      0
Finished            : Mon Jul 13 19:32:13 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 18:36:11 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux o01
TestID              : 0
StartTime           : Mon Jul 13 18:36:11 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy
FS                  : 430.8 TiB   Used FS: 1.9%   Inodes: 2.1 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1
tasks               : 112
clients per node    : 112
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 1059.57 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 5984 max: 26052 -- min data: 11.7 GiB mean data: 27.8 GiB time: 301.2s
WARNING: Expected aggregate file size       = 1165009879040000.
WARNING: Stat() of aggregate file size      = 6119120437248.
WARNING: Using actual aggregate bytes moved = 6119120437248.
WARNING: maybe caused by deadlineForStonewalling
write     10458      5230       0.000111    10158080000 2048.00    0.040895   557.94     0.025561   557.98     0   
Max Write: 10458.48 MiB/sec (10966.52 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       10458.48   10458.48   10458.48       0.00    5229.24    5229.24    5229.24       0.00  557.98216     301.15      10591.74     0    112 112    1   1     1        1         0    0      1 10401873920000  2097152 5835648.0 CEPHFS      0
Finished            : Mon Jul 13 18:45:30 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 19:37:14 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux o01
TestID              : 0
StartTime           : Mon Jul 13 19:37:14 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard
FS                  : 430.5 TiB   Used FS: 3.7%   Inodes: 52.1 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1
tasks               : 112
clients per node    : 112
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 4.79 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 436245 max: 436245 -- min data: 19.1 GiB mean data: 19.1 GiB time: 303.1s
WARNING: Expected aggregate file size       = 5264896000000.
WARNING: Stat() of aggregate file size      = 2296784555520.
WARNING: Using actual aggregate bytes moved = 2296784555520.
read      7224       161161     303.14      45.91      45.91      0.061223   303.17     0.038975   303.21     0   
Max Read:  7223.91 MiB/sec (7574.82 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         7223.91    7223.91    7223.91       0.00  161138.95  161138.95  161138.95       0.00  303.21309     303.13       7225.91     0    112 112    1   0     1        1         0    0 1000000    47008    47008 2190384.5 CEPHFS      0
Finished            : Mon Jul 13 19:42:18 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 18:53:00 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -w -D 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux o01
TestID              : 0
StartTime           : Mon Jul 13 18:53:00 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard
FS                  : 430.6 TiB   Used FS: 3.2%   Inodes: 43.4 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1
tasks               : 112
clients per node    : 112
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 4.79 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 327291 max: 436245 -- min data: 14.3 GiB mean data: 17.8 GiB time: 300.0s
WARNING: Expected aggregate file size       = 5264896000000.
WARNING: Stat() of aggregate file size      = 2296784555520.
WARNING: Using actual aggregate bytes moved = 2296784555520.
WARNING: maybe caused by deadlineForStonewalling
write     5613       125257     300.20      45.91      45.91      0.147916   390.07     0.038964   390.24     0   
Max Write: 5612.96 MiB/sec (5885.62 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        5612.96    5612.96    5612.96       0.00  125204.64  125204.64  125204.64       0.00  390.23665     300.03       6817.85     0    112 112    1   0     1        1         0    0 1000000    47008    47008 2190384.5 CEPHFS      0
Finished            : Mon Jul 13 18:59:31 2020
mdtest-easy-delete
-- started at 07/13/2020 19:44:34 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '400000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy-stonewall' '-r' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.5 TiB   Used FS: 3.7%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 44800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      61975.341      61975.312      61975.331          0.006
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          5.297          5.297          5.297          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        722.868        722.868        722.868          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.189          0.189          0.189          0.000
-- finished at 07/13/2020 19:56:37 --

mdtest-easy-stat
-- started at 07/13/2020 19:32:27 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '400000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy-stonewall' '-T' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.5 TiB   Used FS: 3.7%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 44800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     165348.982     165348.828     165348.873          0.027
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        270.942        270.942        270.942          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 19:36:59 --

mdtest-easy-write
-- started at 07/13/2020 18:45:44 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '400000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.6 TiB   Used FS: 3.2%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 44800000 files
Continue stonewall hit min: 257523 max: 400000 avg: 379868.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     110705.063     110704.856     110704.957          0.048
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     141773.388             NA
   Tree creation             :         21.849         21.849         21.849          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        404.680        404.679        404.679          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.093             NA
   Tree creation             :          0.046          0.046          0.046          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 18:52:28 --

mdtest-hard-delete
-- started at 07/13/2020 19:59:37 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall' '-r' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.5 TiB   Used FS: 3.7%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 11200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       9862.129       9862.126       9862.128          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.574          2.574          2.574          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        577.630        577.629        577.629          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.389          0.389          0.389          0.000
-- finished at 07/13/2020 20:09:16 --

mdtest-hard-read
-- started at 07/13/2020 19:57:09 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.5 TiB   Used FS: 3.7%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 11200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      43050.590      43050.195      43050.565          0.038
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        132.326        132.325        132.325          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 19:59:21 --

mdtest-hard-stat
-- started at 07/13/2020 19:42:33 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall' '-T' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.5 TiB   Used FS: 3.7%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 11200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      53904.366      53904.325      53904.348          0.009
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        105.681        105.681        105.681          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 19:44:19 --

mdtest-hard-write
-- started at 07/13/2020 18:59:47 --

mdtest-3.3.0+dev was launched with 112 total task(s) on 1 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app
FS: 430.5 TiB   Used FS: 3.7%   Inodes: 1.3 Mi   Used Inodes: 100.0%

Nodemap: 1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
112 tasks, 11200000 files
Continue stonewall hit min: 42887 max: 50863 avg: 46445.6 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      16861.041      16860.798      16861.037          0.023
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      17343.488             NA
   Tree creation             :         12.777         12.777         12.777          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        337.864        337.859        337.859          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.935             NA
   Tree creation             :          0.078          0.078          0.078          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 19:05:25 --

result
version         = io500-isc20_v4
config-hash     = EF4BAF31
result-dir      = ./results/2020.07.13-17.30.55-app
; START 2020-07-13 18:35:56


[ior-easy-write]
t_start         = 2020-07-13 18:36:10
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
throughput-stonewall = 625.25
score           = 10.213364
t_delta         = 559.0477
t_end           = 2020-07-13 18:45:30

[mdtest-easy-write]
t_start         = 2020-07-13 18:45:43
exe             = ./mdtest -n 400000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy-stonewall -C -Y -W 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
rate-stonewall  = 141.773388
score           = 110.704940
t_delta         = 405.4338
t_end           = 2020-07-13 18:52:29

[timestamp]
t_start         = 2020-07-13 18:52:44
t_delta         = 0.0026
t_end           = 2020-07-13 18:52:44

[ior-hard-write]
t_start         = 2020-07-13 18:53:00
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -w -D 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
throughput-stonewall = 573.33
score           = 5.481411
t_delta         = 390.6569
t_end           = 2020-07-13 18:59:31

[mdtest-hard-write]
t_start         = 2020-07-13 18:59:47
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall -C -Y -W 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
rate-stonewall  = 17.343488
score           = 16.861038
t_delta         = 338.5948
t_end           = 2020-07-13 19:05:26

[find]
t_start         = 2020-07-13 19:05:40
exe             = ./pfind /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app -newer /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 217696
total-files     = 50496774
score           = 266.313303
t_delta         = 189.7446
t_end           = 2020-07-13 19:08:50

[ior-easy-read]
t_start         = 2020-07-13 19:22:23
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 9.699151
t_delta         = 590.2061
t_end           = 2020-07-13 19:32:13

[mdtest-easy-stat]
t_start         = 2020-07-13 19:32:27
exe             = ./mdtest -n 400000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy-stonewall -T -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 165.348980
t_delta         = 271.7093
t_end           = 2020-07-13 19:36:59

[ior-hard-read]
t_start         = 2020-07-13 19:37:14
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 7.054601
t_delta         = 304.0222
t_end           = 2020-07-13 19:42:18

[mdtest-hard-stat]
t_start         = 2020-07-13 19:42:33
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall -T -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 53.904333
t_delta         = 106.1832
t_end           = 2020-07-13 19:44:19

[mdtest-easy-delete]
t_start         = 2020-07-13 19:44:34
exe             = ./mdtest -n 400000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-easy-stonewall -r -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 61.975327
t_delta         = 723.5306
t_end           = 2020-07-13 19:56:37

[mdtest-hard-read]
t_start         = 2020-07-13 19:57:09
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall -E -X -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 43.050559
t_delta         = 132.9221
t_end           = 2020-07-13 19:59:21

[mdtest-hard-delete]
t_start         = 2020-07-13 19:59:37
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.13-17.30.55-app/mdtest-hard-stonewall -r -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 9.862129
t_delta         = 578.7045
t_end           = 2020-07-13 20:09:16

[SCORE]
MD              = 57.323105
BW              = 7.867139
SCORE           = 21.236027 
hash            = 4968A7E0
; END 2020-07-13 20:09:46
result_summary
IO500 version io500-isc20_v4
[RESULT]       ior-easy-write       10.213364 GiB/s  : time 559.048 seconds
[RESULT]    mdtest-easy-write      110.704940 kIOPS : time 405.434 seconds
[RESULT]       ior-hard-write        5.481411 GiB/s  : time 390.657 seconds
[RESULT]    mdtest-hard-write       16.861038 kIOPS : time 338.595 seconds
[RESULT]                 find      266.313303 kIOPS : time 189.745 seconds
[RESULT]        ior-easy-read        9.699151 GiB/s  : time 590.206 seconds
[RESULT]     mdtest-easy-stat      165.348980 kIOPS : time 271.709 seconds
[RESULT]        ior-hard-read        7.054601 GiB/s  : time 304.022 seconds
[RESULT]     mdtest-hard-stat       53.904333 kIOPS : time 106.183 seconds
[RESULT]   mdtest-easy-delete       61.975327 kIOPS : time 723.531 seconds
[RESULT]     mdtest-hard-read       43.050559 kIOPS : time 132.922 seconds
[RESULT]   mdtest-hard-delete        9.862129 kIOPS : time 578.704 seconds
[SCORE] Bandwidth 7.867139 GB/s : IOPS 57.323105 kiops : TOTAL 21.236027