EC2-10xi3en.metal

Institution Red Hat
Client Procs Per Node
Client Operating System Fedora
Client Operating System Version 31
Client Kernel Version 5.7.6-201.fc32.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 768GB
Storage Interface
Network
Software Version
OS Version Fedora 31

INFORMATION

Client Nodes 10
Client Total Procs 320
Metadata Nodes 10
Metadata Storage Devices 8
Data Nodes 10
Data Storage Devices 8

METADATA

Easy Write 189.58 kIOP/s
Easy Stat 917.71 kIOP/s
Easy Delete 121.26 kIOP/s
Hard Write 20.63 kIOP/s
Hard Read 68.94 kIOP/s
Hard Stat 119.17 kIOP/s
Hard Delete 11.52 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul  7 20:55:25 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux io500-c1
TestID              : 0
StartTime           : Tue Jul  7 20:55:25 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy
FS                  : 49.3 TiB   Used FS: 35.0%   Inodes: 93.1 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 360
clients per node    : 36
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 3405.76 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 3744674611200000.
WARNING: Stat() of aggregate file size      = 14241843118080.
WARNING: Using actual aggregate bytes moved = 14241843118080.
read      56326      28180      0.000049    10158080000 2048.00    0.138071   240.99     0.002234   241.13     0   
Max Read:  56326.44 MiB/sec (59062.56 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        56326.44   56326.44   56326.44       0.00   28163.22   28163.22   28163.22       0.00  241.13150         NA            NA     0    360  36    1   1     1        1         0    0      1 10401873920000  2097152 13582080.0 CEPHFS      0
Finished            : Tue Jul  7 20:59:27 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul  7 20:27:23 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux io500-c1
TestID              : 0
StartTime           : Tue Jul  7 20:27:23 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy
FS                  : 55.2 TiB   Used FS: 0.0%   Inodes: 0.0 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 360
clients per node    : 36
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 3405.76 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 9772 max: 18864 -- min data: 19.1 GiB mean data: 28.4 GiB time: 305.0s
WARNING: Expected aggregate file size       = 3744674611200000.
WARNING: Stat() of aggregate file size      = 14241843118080.
WARNING: Using actual aggregate bytes moved = 14241843118080.
WARNING: maybe caused by deadlineForStonewalling
write     31494      15748      0.000077    10158080000 2048.00    0.029155   431.23     0.000900   431.26     0   
Max Write: 31494.22 MiB/sec (33024.08 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       31494.22   31494.22   31494.22       0.00   15747.11   15747.11   15747.11       0.00  431.25627     304.97      34317.19     0    360  36    1   1     1        1         0    0      1 10401873920000  2097152 13582080.0 CEPHFS      0
Finished            : Tue Jul  7 20:34:35 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul  7 21:01:02 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux io500-c1
TestID              : 0
StartTime           : Tue Jul  7 21:01:02 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard
FS                  : 49.3 TiB   Used FS: 35.0%   Inodes: 93.1 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 360
clients per node    : 36
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 15.39 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 277842 max: 277842 -- min data: 12.2 GiB mean data: 12.2 GiB time: 203.2s
WARNING: Expected aggregate file size       = 16922880000000.
WARNING: Stat() of aggregate file size      = 4701886824960.
WARNING: Using actual aggregate bytes moved = 4701886824960.
read      22065      492288     203.18      45.91      45.91      0.037140   203.18     0.001905   203.22     0   
Max Read:  22065.32 MiB/sec (23137.17 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        22065.32   22065.32   22065.32       0.00  492196.37  492196.37  492196.37       0.00  203.21791     203.17      22070.59     0    360  36    1   0     1        1         0    0 1000000    47008    47008 4484068.5 CEPHFS      0
Finished            : Tue Jul  7 21:04:25 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul  7 20:42:11 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -w -D 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine             : Linux io500-c1
TestID              : 0
StartTime           : Tue Jul  7 20:42:12 2020
Path                : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard
FS                  : 50.8 TiB   Used FS: 25.5%   Inodes: 82.9 Mi   Used Inodes: 100.0%

Options: 
api                 : CEPHFS
apiVersion          : 
test filename       : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 360
clients per node    : 36
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 15.39 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 215011 max: 277842 -- min data: 9.4 GiB mean data: 11.6 GiB time: 300.2s
WARNING: Expected aggregate file size       = 16922880000000.
WARNING: Stat() of aggregate file size      = 4701886824960.
WARNING: Using actual aggregate bytes moved = 4701886824960.
WARNING: maybe caused by deadlineForStonewalling
write     12197      272176     300.36      45.91      45.91      0.139823   367.49     0.001263   367.64     0   
Max Write: 12197.06 MiB/sec (12789.54 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       12197.06   12197.06   12197.06       0.00  272071.59  272071.59  272071.59       0.00  367.63530     300.21      14185.42     0    360  36    1   0     1        1         0    0 1000000    47008    47008 4484068.5 CEPHFS      0
Finished            : Tue Jul  7 20:48:20 2020
mdtest-easy-delete
-- started at 07/07/2020 21:05:22 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '240000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy-stonewall' '-r' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 49.3 TiB   Used FS: 35.0%   Inodes: 88.8 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 86400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     121258.221     121257.954     121258.078          0.053
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          3.476          3.476          3.476          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        712.531        712.529        712.530          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.288          0.288          0.288          0.000
-- finished at 07/07/2020 21:17:15 --

mdtest-easy-stat
-- started at 07/07/2020 20:59:27 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '240000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy-stonewall' '-T' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 49.3 TiB   Used FS: 35.0%   Inodes: 88.8 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 86400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     917719.130     917705.407     917710.613          2.398
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         94.148         94.146         94.147          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/07/2020 21:01:01 --

mdtest-easy-write
-- started at 07/07/2020 20:34:35 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '240000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 50.9 TiB   Used FS: 25.4%   Inodes: 0.0 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 86400000 files
Continue stonewall hit min: 110585 max: 240000 avg: 222666.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     189582.695     189582.285     189582.454          0.080
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     266719.536             NA
   Tree creation             :          8.243          8.243          8.243          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        455.739        455.738        455.738          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.540             NA
   Tree creation             :          0.121          0.121          0.121          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/07/2020 20:42:11 --

mdtest-hard-delete
-- started at 07/07/2020 21:18:53 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall' '-r' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 49.3 TiB   Used FS: 35.0%   Inodes: 88.8 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 36000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      11519.947      11519.906      11519.930          0.007
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         18.811         18.811         18.811          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        582.067        582.065        582.066          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.053          0.053          0.053          0.000
-- finished at 07/07/2020 21:28:36 --

mdtest-hard-read
-- started at 07/07/2020 21:17:16 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 49.3 TiB   Used FS: 35.0%   Inodes: 88.8 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 36000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      68942.853      68941.830      68942.362          0.204
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         97.261         97.260         97.260          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/07/2020 21:18:53 --

mdtest-hard-stat
-- started at 07/07/2020 21:04:25 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall' '-T' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 49.3 TiB   Used FS: 35.0%   Inodes: 88.8 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 36000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     119173.042     119169.891     119171.465          0.476
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         56.267         56.266         56.266          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/07/2020 21:05:22 --

mdtest-hard-write
-- started at 07/07/2020 20:48:20 --

mdtest-3.3.0+dev was launched with 360 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app
FS: 49.3 TiB   Used FS: 35.0%   Inodes: 82.4 Mi   Used Inodes: 100.0%

Nodemap: 111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 36 for each phase.
360 tasks, 36000000 files
Continue stonewall hit min: 14912 max: 18626 avg: 16865.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      20631.415      20631.364      20631.387          0.009
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      20110.220             NA
   Tree creation             :        547.490        547.490        547.490          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        325.008        325.007        325.008          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.914             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/07/2020 20:53:45 --

result
version         = io500-isc20_v4
config-hash     = CE64F3BB
result-dir      = /home/fedora/io500-app/results/2020.07.07-20.27.19-app
; START 2020-07-07 20:27:23


[ior-easy-write]
t_start         = 2020-07-07 20:27:23
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
throughput-stonewall = 8531.70
score           = 30.756075
t_delta         = 431.6146
t_end           = 2020-07-07 20:34:35

[mdtest-easy-write]
t_start         = 2020-07-07 20:34:35
exe             = ./mdtest -n 240000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy-stonewall -C -Y -W 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
rate-stonewall  = 266.719536
score           = 189.582331
t_delta         = 456.6563
t_end           = 2020-07-07 20:42:11

[timestamp]
t_start         = 2020-07-07 20:42:11
t_delta         = 0.0005
t_end           = 2020-07-07 20:42:11

[ior-hard-write]
t_start         = 2020-07-07 20:42:11
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -w -D 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
throughput-stonewall = 4072.38
score           = 11.911189
t_delta         = 368.3363
t_end           = 2020-07-07 20:48:20

[mdtest-hard-write]
t_start         = 2020-07-07 20:48:20
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall -C -Y -W 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
rate-stonewall  = 20.110220
score           = 20.631366
t_delta         = 325.8275
t_end           = 2020-07-07 20:53:46

[find]
t_start         = 2020-07-07 20:53:46
exe             = ./pfind /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app -newer /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 221883
total-files     = 93105726
score           = 938.666920
t_delta         = 99.1907
t_end           = 2020-07-07 20:55:25

[ior-easy-read]
t_start         = 2020-07-07 20:55:25
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 55.006292
t_delta         = 241.9220
t_end           = 2020-07-07 20:59:27

[mdtest-easy-stat]
t_start         = 2020-07-07 20:59:27
exe             = ./mdtest -n 240000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy-stonewall -T -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 917.709973
t_delta         = 94.7418
t_end           = 2020-07-07 21:01:02

[ior-hard-read]
t_start         = 2020-07-07 21:01:02
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 21.548166
t_delta         = 203.4786
t_end           = 2020-07-07 21:04:25

[mdtest-hard-stat]
t_start         = 2020-07-07 21:04:25
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall -T -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 119.171335
t_delta         = 56.8966
t_end           = 2020-07-07 21:05:22

[mdtest-easy-delete]
t_start         = 2020-07-07 21:05:22
exe             = ./mdtest -n 240000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-easy-stonewall -r -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 121.258101
t_delta         = 713.0408
t_end           = 2020-07-07 21:17:15

[mdtest-hard-read]
t_start         = 2020-07-07 21:17:15
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall -E -X -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 68.942548
t_delta         = 97.6661
t_end           = 2020-07-07 21:18:53

[mdtest-hard-delete]
t_start         = 2020-07-07 21:18:53
exe             = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.07-20.27.19-app/mdtest-hard-stonewall -r -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score           = 11.519936
t_delta         = 582.5288
t_end           = 2020-07-07 21:28:36

[SCORE]
MD              = 118.418715
BW              = 25.670077
SCORE           = 55.134541 
hash            = F9CE3A4
; END 2020-07-07 21:29:50
result_summary
IO500 version io500-isc20_v4
[RESULT]       ior-easy-write       30.756075 GiB/s  : time 431.615 seconds
[RESULT]    mdtest-easy-write      189.582331 kIOPS : time 456.656 seconds
[RESULT]       ior-hard-write       11.911189 GiB/s  : time 368.336 seconds
[RESULT]    mdtest-hard-write       20.631366 kIOPS : time 325.828 seconds
[RESULT]                 find      938.666920 kIOPS : time 99.191 seconds
[RESULT]        ior-easy-read       55.006292 GiB/s  : time 241.922 seconds
[RESULT]     mdtest-easy-stat      917.709973 kIOPS : time 94.742 seconds
[RESULT]        ior-hard-read       21.548166 GiB/s  : time 203.479 seconds
[RESULT]     mdtest-hard-stat      119.171335 kIOPS : time 56.897 seconds
[RESULT]   mdtest-easy-delete      121.258101 kIOPS : time 713.041 seconds
[RESULT]     mdtest-hard-read       68.942548 kIOPS : time 97.666 seconds
[RESULT]   mdtest-hard-delete       11.519936 kIOPS : time 582.529 seconds
[SCORE] Bandwidth 25.670077 GB/s : IOPS 118.418715 kiops : TOTAL 55.134541