HammerHead

Institution SUSE
Client Procs Per Node
Client Operating System SUSE
Client Operating System Version SLES 15 SP 1
Client Kernel Version 4.12.14-197.37-default

DATA SERVER

Storage Type NVMe
Volatile Memory 128G
Storage Interface Ethernet
Network Ethernet
Software Version 6
OS Version SLES 15 SP 1

INFORMATION

Client Nodes 10
Client Total Procs 220
Metadata Nodes 12
Metadata Storage Devices 4
Data Nodes 10
Data Storage Devices 4

METADATA

Easy Write 21.39 kIOP/s
Easy Stat 93.87 kIOP/s
Easy Delete 18.17 kIOP/s
Hard Write 8.84 kIOP/s
Hard Read 28.06 kIOP/s
Hard Stat 103.75 kIOP/s
Hard Delete 5.63 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 10:28:52 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Sat Jul 11 10:28:52 2020
Path                : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy
FS                  : 62.8 TiB   Used FS: 19.5%   Inodes: 22.4 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 220
clients per node    : 22
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 2081.30 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 2288412262400000.
WARNING: Stat() of aggregate file size      = 5357007011840.
WARNING: Using actual aggregate bytes moved = 5357007011840.
read      7594       3802       0.000135    10158080000 2048.00    0.973166   671.80     0.003532   672.77     0   
Max Read:  7593.74 MiB/sec (7962.61 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         7593.74    7593.74    7593.74       0.00    3796.87    3796.87    3796.87       0.00  672.77003         NA            NA     0    220  22    1   1     1        1         0    0      1 10401873920000  2097152 5108840.0 POSIX      0
Finished            : Sat Jul 11 10:40:05 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 10:03:08 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Sat Jul 11 10:03:08 2020
Path                : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy
FS                  : 64.6 TiB   Used FS: 9.9%   Inodes: 9.6 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 220
clients per node    : 22
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 2081.30 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 7941 max: 11611 -- min data: 15.5 GiB mean data: 18.4 GiB time: 305.6s
WARNING: Expected aggregate file size       = 2288412262400000.
WARNING: Stat() of aggregate file size      = 5357007011840.
WARNING: Using actual aggregate bytes moved = 5357007011840.
WARNING: maybe caused by deadlineForStonewalling
write     12965      6483       0.000065    10158080000 2048.00    0.030443   394.03     0.000959   394.06     0   
Max Write: 12964.56 MiB/sec (13594.33 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       12964.56   12964.56   12964.56       0.00    6482.28    6482.28    6482.28       0.00  394.06195     305.59      13573.86     0    220  22    1   1     1        1         0    0      1 10401873920000  2097152 5108840.0 POSIX      0
Finished            : Sat Jul 11 10:09:42 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 10:41:39 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -r -R -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Sat Jul 11 10:41:39 2020
Path                : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard
FS                  : 62.8 TiB   Used FS: 19.5%   Inodes: 22.4 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 500000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 220
clients per node    : 22
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 4.70 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 102101 max: 102101 -- min data: 4.5 GiB mean data: 4.5 GiB time: 80.6s
WARNING: Expected aggregate file size       = 5170880000000.
WARNING: Stat() of aggregate file size      = 1055904037760.
WARNING: Using actual aggregate bytes moved = 1055904037760.
read      12491      278656     80.61       45.91      45.91      0.005093   80.61      0.003645   80.61      0   
Max Read:  12491.37 MiB/sec (13098.15 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        12491.37   12491.37   12491.37       0.00  278636.65  278636.65  278636.65       0.00   80.61474      80.60      12493.39     0    220  22    1   0     1        1         0    0 500000    47008    47008 1006988.6 POSIX      0
Finished            : Sat Jul 11 10:42:59 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 10:16:33 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -w -D 300 -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Sat Jul 11 10:16:33 2020
Path                : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard
FS                  : 63.1 TiB   Used FS: 17.8%   Inodes: 19.3 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 500000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 220
clients per node    : 22
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 4.70 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 78333 max: 102101 -- min data: 3.4 GiB mean data: 3.9 GiB time: 300.4s
WARNING: Expected aggregate file size       = 5170880000000.
WARNING: Stat() of aggregate file size      = 1055904037760.
WARNING: Using actual aggregate bytes moved = 1055904037760.
WARNING: maybe caused by deadlineForStonewalling
write     2592.02    57819      302.76      45.91      45.91      0.003211   388.49     0.001831   388.50     0   
Max Write: 2592.02 MiB/sec (2717.93 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2592.02    2592.02    2592.02       0.00   57818.53   57818.53   57818.53       0.00  388.49519     300.37       2951.78     0    220  22    1   0     1        1         0    0 500000    47008    47008 1006988.6 POSIX      0
Finished            : Sat Jul 11 10:23:02 2020
mdtest-easy-delete
-- started at 07/11/2020 10:43:26 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 62.8 TiB   Used FS: 19.5%   Inodes: 22.4 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      18172.849      18172.802      18172.831          0.015
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.508          0.508          0.508          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        484.240        484.239        484.239          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.967          1.967          1.967          0.000
-- finished at 07/11/2020 10:51:33 --

mdtest-easy-stat
-- started at 07/11/2020 10:40:05 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 62.8 TiB   Used FS: 19.5%   Inodes: 22.4 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      93869.722      93869.496      93869.632          0.067
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         93.747         93.747         93.747          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 10:41:39 --

mdtest-easy-write
-- started at 07/11/2020 10:09:42 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 63.1 TiB   Used FS: 17.8%   Inodes: 10.8 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files
Continue stonewall hit min: 14679 max: 40000 avg: 34444.9 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      21386.048      21385.998      21386.028          0.016
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      25138.048             NA
   Tree creation             :         25.235         25.235         25.235          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        411.484        411.483        411.484          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.451             NA
   Tree creation             :          0.040          0.040          0.040          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 10:16:33 --

mdtest-hard-delete
-- started at 07/11/2020 10:53:10 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 62.8 TiB   Used FS: 19.5%   Inodes: 14.0 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       5628.653       5628.640       5628.647          0.004
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         15.853         15.853         15.853          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        484.508        484.507        484.507          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.063          0.063          0.063          0.000
-- finished at 07/11/2020 11:01:15 --

mdtest-hard-read
-- started at 07/11/2020 10:51:33 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 62.8 TiB   Used FS: 19.5%   Inodes: 14.1 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      28059.428      28059.345      28059.390          0.020
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         97.191         97.191         97.191          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 10:53:10 --

mdtest-hard-stat
-- started at 07/11/2020 10:42:59 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 62.8 TiB   Used FS: 19.5%   Inodes: 22.4 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     103752.252     103751.521     103751.994          0.175
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         26.285         26.285         26.285          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 10:43:26 --

mdtest-hard-write
-- started at 07/11/2020 10:23:02 --

mdtest-3.3.0+dev was launched with 220 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.11-10.09.37-app
FS: 62.8 TiB   Used FS: 19.4%   Inodes: 19.8 Mi   Used Inodes: 100.0%

Nodemap: 1000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
220 tasks, 8800000 files
Continue stonewall hit min: 11999 max: 12396 avg: 12141.9 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       8837.608       8837.588       8837.600          0.007
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       8899.480             NA
   Tree creation             :       3761.708       3761.708       3761.708          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        308.582        308.581        308.582          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.155             NA
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 10:28:11 --

result
version         = 
config-hash     = 7392AFA9
result-dir      = /mnt/cephfs/results/2020.07.11-10.09.37-app
; START 2020-07-11 10:03:08


[ior-easy-write]
t_start         = 2020-07-11 10:03:08
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 2261.53
score           = 12.660704
t_delta         = 394.1956
t_end           = 2020-07-11 10:09:42

[mdtest-easy-write]
t_start         = 2020-07-11 10:09:42
exe             = ./mdtest -n 40000 -u -L -F -P -N 1 -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 25.138048
score           = 21.386033
t_delta         = 411.5694
t_end           = 2020-07-11 10:16:33

[timestamp]
t_start         = 2020-07-11 10:16:33
t_delta         = 0.0185
t_end           = 2020-07-11 10:16:33

[ior-hard-write]
t_start         = 2020-07-11 10:16:33
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -w -D 300 -a POSIX
throughput-stonewall = 490.31
score           = 2.531273
t_delta         = 388.5754
t_end           = 2020-07-11 10:23:02

[mdtest-hard-write]
t_start         = 2020-07-11 10:23:02
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 8.899480
score           = 8.837602
t_delta         = 308.6123
t_end           = 2020-07-11 10:28:11

[find]
t_start         = 2020-07-11 10:28:11
exe             = ./pfind /mnt/cephfs/datafiles/2020.07.11-10.09.37-app -newer /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 99348
total-files     = 11527346
score           = 280.433965
t_delta         = 41.1122
t_end           = 2020-07-11 10:28:52

[ior-easy-read]
t_start         = 2020-07-11 10:28:52
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX
score           = 7.415761
t_delta         = 672.9641
t_end           = 2020-07-11 10:40:05

[mdtest-easy-stat]
t_start         = 2020-07-11 10:40:05
exe             = ./mdtest -n 40000 -u -L -F -P -N 1 -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy-stonewall -T -a POSIX
score           = 93.869669
t_delta         = 93.8725
t_end           = 2020-07-11 10:41:39

[ior-hard-read]
t_start         = 2020-07-11 10:41:39
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.11-10.09.37-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -r -R -a POSIX
score           = 12.198604
t_delta         = 80.6441
t_end           = 2020-07-11 10:42:59

[mdtest-hard-stat]
t_start         = 2020-07-11 10:42:59
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall -T -a POSIX
score           = 103.752169
t_delta         = 26.2984
t_end           = 2020-07-11 10:43:26

[mdtest-easy-delete]
t_start         = 2020-07-11 10:43:26
exe             = ./mdtest -n 40000 -u -L -F -P -N 1 -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-easy-stonewall -r -a POSIX
score           = 18.172835
t_delta         = 487.0295
t_end           = 2020-07-11 10:51:33

[mdtest-hard-read]
t_start         = 2020-07-11 10:51:33
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 28.059395
t_delta         = 97.2110
t_end           = 2020-07-11 10:53:10

[mdtest-hard-delete]
t_start         = 2020-07-11 10:53:10
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.11-10.09.37-app/mdtest-hard-stonewall -r -a POSIX
score           = 5.628648
t_delta         = 484.7719
t_end           = 2020-07-11 11:01:15

[SCORE]
MD              = 33.215467
BW              = 7.337796
SCORE           = 15.611801 
hash            = C560138E
; END 2020-07-11 11:01:26
result_summary
IO500 version 
[RESULT]       ior-easy-write       12.660704 GiB/s  : time 394.196 seconds
[RESULT]    mdtest-easy-write       21.386033 kIOPS : time 411.569 seconds
[RESULT]       ior-hard-write        2.531273 GiB/s  : time 388.575 seconds
[RESULT]    mdtest-hard-write        8.837602 kIOPS : time 308.612 seconds
[RESULT]                 find      280.433965 kIOPS : time 41.112 seconds
[RESULT]        ior-easy-read        7.415761 GiB/s  : time 672.964 seconds
[RESULT]     mdtest-easy-stat       93.869669 kIOPS : time 93.873 seconds
[RESULT]        ior-hard-read       12.198604 GiB/s  : time 80.644 seconds
[RESULT]     mdtest-hard-stat      103.752169 kIOPS : time 26.298 seconds
[RESULT]   mdtest-easy-delete       18.172835 kIOPS : time 487.029 seconds
[RESULT]     mdtest-hard-read       28.059395 kIOPS : time 97.211 seconds
[RESULT]   mdtest-hard-delete        5.628648 kIOPS : time 484.772 seconds
[SCORE] Bandwidth 7.337796 GB/s : IOPS 33.215467 kiops : TOTAL 15.611801