HammerHead

Institution SUSE
Client Procs Per Node
Client Operating System SUSE
Client Operating System Version SLES 15 SP 1
Client Kernel Version 4.12.14-197.37-default

DATA SERVER

Storage Type NVMe
Volatile Memory 128G
Storage Interface Ethernet
Network Ethernet
Software Version 6
OS Version SLES 15 SP 1

INFORMATION

Client Nodes 10
Client Total Procs 200
Metadata Nodes 12
Metadata Storage Devices 4
Data Nodes 10
Data Storage Devices 4

METADATA

Easy Write 21.66 kIOP/s
Easy Stat 72.11 kIOP/s
Easy Delete 17.68 kIOP/s
Hard Write 10.04 kIOP/s
Hard Read 25.30 kIOP/s
Hard Stat 80.11 kIOP/s
Hard Delete 6.24 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul  6 03:52:46 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Mon Jul  6 03:52:46 2020
Path                : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy
FS                  : 59.4 TiB   Used FS: 35.8%   Inodes: 16.0 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 200
clients per node    : 20
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 1892.09 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 2080374784000000.
WARNING: Stat() of aggregate file size      = 7250273894400.
WARNING: Using actual aggregate bytes moved = 7250273894400.
read      7110       3597.14    0.000194    10158080000 2048.00    11.41      961.10     0.004876   972.50     0   
Max Read:  7109.89 MiB/sec (7455.26 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         7109.89    7109.89    7109.89       0.00    3554.94    3554.94    3554.94       0.00  972.50490         NA            NA     0    200  20    1   1     1        1         0    0      1 10401873920000  2097152 6914400.0 POSIX      0
Finished            : Mon Jul  6 04:08:59 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul  6 03:24:55 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Mon Jul  6 03:24:55 2020
Path                : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy
FS                  : 61.9 TiB   Used FS: 22.2%   Inodes: 3.5 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 200
clients per node    : 20
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 1892.09 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 8070 max: 17286 -- min data: 15.8 GiB mean data: 21.2 GiB time: 300.1s
WARNING: Expected aggregate file size       = 2080374784000000.
WARNING: Stat() of aggregate file size      = 7250273894400.
WARNING: Using actual aggregate bytes moved = 7250273894400.
WARNING: maybe caused by deadlineForStonewalling
write     12629      6315       0.000067    10158080000 2048.00    0.030491   547.45     0.001567   547.48     0   
Max Write: 12629.45 MiB/sec (13242.93 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       12629.45   12629.45   12629.45       0.00    6314.72    6314.72    6314.72       0.00  547.48243     300.06      14502.86     0    200  20    1   1     1        1         0    0      1 10401873920000  2097152 6914400.0 POSIX      0
Finished            : Mon Jul  6 03:34:02 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul  6 04:10:50 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -r -R -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Mon Jul  6 04:10:50 2020
Path                : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard
FS                  : 59.4 TiB   Used FS: 35.8%   Inodes: 16.0 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 500000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 200
clients per node    : 20
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 4.28 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 105421 max: 105421 -- min data: 4.6 GiB mean data: 4.6 GiB time: 81.2s
WARNING: Expected aggregate file size       = 4700800000000.
WARNING: Stat() of aggregate file size      = 991126073600.
WARNING: Using actual aggregate bytes moved = 991126073600.
read      11640      259661     81.19       45.91      45.91      0.006247   81.20      0.005078   81.20      0   
Max Read:  11640.42 MiB/sec (12205.86 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        11640.42   11640.42   11640.42       0.00  259654.96  259654.96  259654.96       0.00   81.20084      81.19      11641.76     0    200  20    1   0     1        1         0    0 500000    47008    47008  945211.5 POSIX      0
Finished            : Mon Jul  6 04:12:11 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul  6 03:40:12 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -w -D 300 -a POSIX
Machine             : Linux sr650-1
TestID              : 0
StartTime           : Mon Jul  6 03:40:12 2020
Path                : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard
FS                  : 59.8 TiB   Used FS: 34.1%   Inodes: 12.1 Mi   Used Inodes: 100.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 500000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 200
clients per node    : 20
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 4.28 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 80638 max: 105421 -- min data: 3.5 GiB mean data: 4.0 GiB time: 300.0s
WARNING: Expected aggregate file size       = 4700800000000.
WARNING: Stat() of aggregate file size      = 991126073600.
WARNING: Using actual aggregate bytes moved = 991126073600.
WARNING: maybe caused by deadlineForStonewalling
write     2459.16    54855      366.26      45.91      45.91      0.005118   384.36     0.002321   384.36     0   
Max Write: 2459.16 MiB/sec (2578.62 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2459.16    2459.16    2459.16       0.00   54854.84   54854.84   54854.84       0.00  384.36354     300.03       2762.61     0    200  20    1   0     1        1         0    0 500000    47008    47008  945211.5 POSIX      0
Finished            : Mon Jul  6 03:46:36 2020
mdtest-easy-delete
-- started at 07/06/2020 04:12:51 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.4 TiB   Used FS: 35.8%   Inodes: 16.0 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      17678.094      17678.047      17678.076          0.014
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.449          0.449          0.449          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        452.539        452.537        452.538          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.225          2.225          2.225          0.000
-- finished at 07/06/2020 04:20:26 --

mdtest-easy-stat
-- started at 07/06/2020 04:08:59 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.4 TiB   Used FS: 35.8%   Inodes: 16.0 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      72108.706      72108.466      72108.623          0.071
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        110.944        110.944        110.944          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/06/2020 04:10:50 --

mdtest-easy-write
-- started at 07/06/2020 03:34:02 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.8 TiB   Used FS: 34.0%   Inodes: 5.1 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files
Continue stonewall hit min: 12040 max: 40000 avg: 35924.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      21660.028      21659.976      21660.007          0.016
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      23856.688             NA
   Tree creation             :         30.951         30.951         30.951          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        369.345        369.344        369.344          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.172             NA
   Tree creation             :          0.032          0.032          0.032          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/06/2020 03:40:11 --

mdtest-hard-delete
-- started at 07/06/2020 04:22:32 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.4 TiB   Used FS: 35.8%   Inodes: 8.4 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       6243.579       6243.564       6243.573          0.005
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          5.588          5.588          5.588          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        510.157        510.156        510.157          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.179          0.179          0.179          0.000
-- finished at 07/06/2020 04:31:03 --

mdtest-hard-read
-- started at 07/06/2020 04:20:26 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.4 TiB   Used FS: 35.8%   Inodes: 8.4 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      25299.806      25299.734      25299.774          0.020
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        125.899        125.898        125.898          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/06/2020 04:22:32 --

mdtest-hard-stat
-- started at 07/06/2020 04:12:11 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.4 TiB   Used FS: 35.8%   Inodes: 16.0 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      80112.173      80111.923      80112.071          0.060
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         39.759         39.759         39.759          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/06/2020 04:12:51 --

mdtest-hard-write
-- started at 07/06/2020 03:46:36 --

mdtest-3.3.0+dev was launched with 200 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '40000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard' '-x' '/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/cephfs/datafiles/2020.07.06-02.27.35-app
FS: 59.4 TiB   Used FS: 35.8%   Inodes: 12.7 Mi   Used Inodes: 100.0%

Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
200 tasks, 8000000 files
Continue stonewall hit min: 15226 max: 15926 avg: 15412.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      10035.873      10035.851      10035.865          0.007
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      10266.795             NA
   Tree creation             :       4593.980       4593.980       4593.980          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        317.382        317.381        317.382          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.232             NA
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/06/2020 03:51:53 --

result
version         = 
config-hash     = 7392AFA9
result-dir      = /mnt/cephfs/results/2020.07.06-02.27.35-app
; START 2020-07-06 03:24:55


[ior-easy-write]
t_start         = 2020-07-06 03:24:55
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 1552.48
score           = 12.333444
t_delta         = 547.5394
t_end           = 2020-07-06 03:34:02

[mdtest-easy-write]
t_start         = 2020-07-06 03:34:02
exe             = ./mdtest -n 40000 -u -L -F -P -N 1 -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 23.856688
score           = 21.660016
t_delta         = 369.4114
t_end           = 2020-07-06 03:40:11

[timestamp]
t_start         = 2020-07-06 03:40:11
t_delta         = 0.0730
t_end           = 2020-07-06 03:40:12

[ior-hard-write]
t_start         = 2020-07-06 03:40:12
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -w -D 300 -a POSIX
throughput-stonewall = 421.19
score           = 2.401524
t_delta         = 384.3969
t_end           = 2020-07-06 03:46:36

[mdtest-hard-write]
t_start         = 2020-07-06 03:46:36
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 10.266795
score           = 10.035869
t_delta         = 317.4293
t_end           = 2020-07-06 03:51:53

[find]
t_start         = 2020-07-06 03:51:53
exe             = ./pfind /mnt/cephfs/datafiles/2020.07.06-02.27.35-app -newer /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 97118
total-files     = 11185406
score           = 211.432501
t_delta         = 52.9595
t_end           = 2020-07-06 03:52:46

[ior-easy-read]
t_start         = 2020-07-06 03:52:46
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX
score           = 6.943249
t_delta         = 972.5944
t_end           = 2020-07-06 04:08:59

[mdtest-easy-stat]
t_start         = 2020-07-06 04:08:59
exe             = ./mdtest -n 40000 -u -L -F -P -N 1 -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy-stonewall -T -a POSIX
score           = 72.108635
t_delta         = 111.1364
t_end           = 2020-07-06 04:10:50

[ior-hard-read]
t_start         = 2020-07-06 04:10:50
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/file -O stoneWallingStatusFile=/mnt/cephfs/datafiles/2020.07.06-02.27.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 500000 -r -R -a POSIX
score           = 11.367593
t_delta         = 81.2339
t_end           = 2020-07-06 04:12:11

[mdtest-hard-stat]
t_start         = 2020-07-06 04:12:11
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall -T -a POSIX
score           = 80.112123
t_delta         = 39.7893
t_end           = 2020-07-06 04:12:51

[mdtest-easy-delete]
t_start         = 2020-07-06 04:12:51
exe             = ./mdtest -n 40000 -u -L -F -P -N 1 -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-easy-stonewall -r -a POSIX
score           = 17.678079
t_delta         = 455.1802
t_end           = 2020-07-06 04:20:26

[mdtest-hard-read]
t_start         = 2020-07-06 04:20:26
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 25.299773
t_delta         = 125.9400
t_end           = 2020-07-06 04:22:32

[mdtest-hard-delete]
t_start         = 2020-07-06 04:22:32
exe             = ./mdtest -n 40000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard -x /mnt/cephfs/datafiles/2020.07.06-02.27.35-app/mdtest-hard-stonewall -r -a POSIX
score           = 6.243575
t_delta         = 510.7514
t_end           = 2020-07-06 04:31:03

[SCORE]
MD              = 30.461901
BW              = 6.953455
SCORE           = 14.553881 
hash            = A86C8F24
; END 2020-07-06 04:31:13
result_summary
IO500 version 
[RESULT]       ior-easy-write       12.333444 GiB/s  : time 547.539 seconds
[RESULT]    mdtest-easy-write       21.660016 kIOPS : time 369.411 seconds
[RESULT]       ior-hard-write        2.401524 GiB/s  : time 384.397 seconds
[RESULT]    mdtest-hard-write       10.035869 kIOPS : time 317.429 seconds
[RESULT]                 find      211.432501 kIOPS : time 52.959 seconds
[RESULT]        ior-easy-read        6.943249 GiB/s  : time 972.594 seconds
[RESULT]     mdtest-easy-stat       72.108635 kIOPS : time 111.136 seconds
[RESULT]        ior-hard-read       11.367593 GiB/s  : time 81.234 seconds
[RESULT]     mdtest-hard-stat       80.112123 kIOPS : time 39.789 seconds
[RESULT]   mdtest-easy-delete       17.678079 kIOPS : time 455.180 seconds
[RESULT]     mdtest-hard-read       25.299773 kIOPS : time 125.940 seconds
[RESULT]   mdtest-hard-delete        6.243575 kIOPS : time 510.751 seconds
[SCORE] Bandwidth 6.953455 GB/s : IOPS 30.461901 kiops : TOTAL 14.553881