Mistral

Institution German Climate Computing Center
Client Procs Per Node
Client Operating System RHEL
Client Operating System Version 6.10
Client Kernel Version 2.6.32-754.14.2.el6.x86_64

DATA SERVER

Storage Type HDD
Volatile Memory
Storage Interface
Network Infiniband
Software Version
OS Version

INFORMATION

Client Nodes 10
Client Total Procs 120
Metadata Nodes 1
Metadata Storage Devices 0
Data Nodes 62
Data Storage Devices 5,208

METADATA

Easy Write 13.63 kIOP/s
Easy Stat 37.68 kIOP/s
Easy Delete 7.07 kIOP/s
Hard Write 9.68 kIOP/s
Hard Read 10.70 kIOP/s
Hard Stat 29.89 kIOP/s
Hard Delete 8.05 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue May 19 12:40:00 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX
Machine             : Linux m11404
TestID              : 0
StartTime           : Tue May 19 12:40:00 2020
Path                : /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app/ior-easy
FS                  : 20835.6 TiB   Used FS: 79.6%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 120
clients per node    : 12
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 1135.25 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 1248224870400000.
WARNING: Stat() of aggregate file size      = 15794574458880.
WARNING: Using actual aggregate bytes moved = 15794574458880.
read      26346      13179      0.000115    10158080000 2048.00    0.015176   571.47     0.192034   571.74     0   
Max Read:  26345.74 MiB/sec (27625.51 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        26345.74   26345.74   26345.74       0.00   13172.87   13172.87   13172.87       0.00  571.73866         NA            NA     0    120  12    1   1     1        1         0    0      1 10401873920000  2097152 15062880.0 POSIX      0
Finished            : Tue May 19 12:49:32 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue May 19 12:12:50 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux m11404
TestID              : 0
StartTime           : Tue May 19 12:12:50 2020
Path                : /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app/ior-easy
FS                  : 20835.6 TiB   Used FS: 79.6%   Inodes: 5974.4 Mi   Used Inodes: 8.3%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 120
clients per node    : 12
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 1135.25 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 29552 max: 62762 -- min data: 57.7 GiB mean data: 88.0 GiB time: 300.9s
WARNING: Expected aggregate file size       = 1248224870400000.
WARNING: Stat() of aggregate file size      = 15794574458880.
WARNING: Using actual aggregate bytes moved = 15794574458880.
WARNING: maybe caused by deadlineForStonewalling
write     31519      15761      0.000080    10158080000 2048.00    0.026540   477.86     0.012107   477.90     0   
Max Write: 31518.84 MiB/sec (33049.90 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       31518.84   31518.84   31518.84       0.00   15759.42   15759.42   15759.42       0.00  477.90077     300.95      35933.83     0    120  12    1   1     1        1         0    0      1 10401873920000  2097152 15062880.0 POSIX      0
Finished            : Tue May 19 12:20:48 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue May 19 12:51:29 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/file -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 8500 -r -R -a POSIX
Machine             : Linux m11404
TestID              : 0
StartTime           : Tue May 19 12:51:29 2020
Path                : /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app/ior-hard
FS                  : 20835.6 TiB   Used FS: 79.5%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 8500
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 120
clients per node    : 12
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 44.66 GiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 8500 max: 8500 -- min data: 0.4 GiB mean data: 0.4 GiB time: 56.6s
read      807.94     18025      56.59       45.91      45.91      0.007818   56.59      0.000718   56.60      0   
Max Read:  807.94 MiB/sec (847.19 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read          807.94     807.94     807.94       0.00   18022.16   18022.16   18022.16       0.00   56.59700      56.59        808.06     0    120  12    1   0     1        1         0    0   8500    47008    47008   45726.9 POSIX      0
Finished            : Tue May 19 12:52:25 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue May 19 12:26:10 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/file -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 8500 -w -D 300 -a POSIX
Machine             : Linux m11404
TestID              : 0
StartTime           : Tue May 19 12:26:10 2020
Path                : /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app/ior-hard
FS                  : 20835.6 TiB   Used FS: 79.6%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 8500
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 120
clients per node    : 12
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 44.66 GiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 6097 max: 8500 -- min data: 0.3 GiB mean data: 0.4 GiB time: 300.3s
write     113.92     2541.37    300.47      45.91      45.91      0.025299   401.36     0.000735   401.38     0   
Max Write: 113.92 MiB/sec (119.46 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write         113.92     113.92     113.92       0.00    2541.21    2541.21    2541.21       0.00  401.38333     300.33        150.82     0    120  12    1   0     1        1         0    0   8500    47008    47008   45726.9 POSIX      0
Finished            : Tue May 19 12:32:52 2020
mdtest-easy-delete
-- started at 05/19/2020 12:54:06 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.5%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       7066.003       7066.000       7066.002          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          5.850          5.850          5.850          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        621.364        621.364        621.364          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.171          0.171          0.171          0.000
-- finished at 05/19/2020 13:04:28 --

mdtest-easy-stat
-- started at 05/19/2020 12:49:32 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.5%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      37680.542      37680.293      37680.323          0.069
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        116.521        116.521        116.521          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 05/19/2020 12:51:29 --

mdtest-easy-write
-- started at 05/19/2020 12:20:48 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy-stonewall' '-C' '-W' '300' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.6%   Inodes: 5974.4 Mi   Used Inodes: 8.3%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files
Continue stonewall hit min: 34048 max: 36588 avg: 34829.9 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      13635.026      13632.325      13633.341          0.724
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      13865.753             NA
   Tree creation             :         10.048         10.048         10.048          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        322.070        322.006        322.046          0.017
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.432             NA
   Tree creation             :          0.100          0.100          0.100          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 05/19/2020 12:26:10 --

mdtest-hard-delete
-- started at 05/19/2020 13:09:09 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.3%   Inodes: 5974.4 Mi   Used Inodes: 8.3%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       8049.258       8049.000       8049.231          0.077
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.430          1.430          1.430          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        373.224        373.212        373.213          0.004
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.699          0.699          0.699          0.000
-- finished at 05/19/2020 13:15:23 --

mdtest-hard-read
-- started at 05/19/2020 13:04:28 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.4%   Inodes: 5974.4 Mi   Used Inodes: 8.3%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      10696.823      10696.817      10696.820          0.002
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        280.839        280.839        280.839          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 05/19/2020 13:09:09 --

mdtest-hard-stat
-- started at 05/19/2020 12:52:25 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.5%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      29886.616      29886.606      29886.611          0.003
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        100.516        100.516        100.516          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 05/19/2020 12:54:06 --

mdtest-hard-write
-- started at 05/19/2020 12:32:52 --

mdtest-3.3.0+dev was launched with 120 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard' '-x' '/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall' '-C' '-W' '300' '-a' 'POSIX'
Path: /mnt/lustre01/work/k20200/k202079/io500-data/2020.05.19-12.12.45-app
FS: 20835.6 TiB   Used FS: 79.6%   Inodes: 5974.4 Mi   Used Inodes: 8.4%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
120 tasks, 120000000 files
Continue stonewall hit min: 23681 max: 25034 avg: 24270.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       9683.825       9683.822       9683.823          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       9707.868             NA
   Tree creation             :       2949.581       2949.581       2949.581          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        310.216        310.216        310.216          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.008             NA
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 05/19/2020 12:38:02 --

result
version         = dev-17-gd9f8a8406c74
config-hash     = E53CC38C
result-dir      = ./results/2020.05.19-12.12.45-app
; START 2020-05-19 12:12:50


[ior-easy-write]
t_start         = 2020-05-19 12:12:50
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 2651.79
score           = 30.780122
t_delta         = 478.1561
t_end           = 2020-05-19 12:20:48

[mdtest-easy-write]
t_start         = 2020-05-19 12:20:48
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy-stonewall -C -W 300 -a POSIX
rate-stonewall  = 13.865753
score           = 13.633290
t_delta         = 322.3377
t_end           = 2020-05-19 12:26:10

[timestamp]
t_start         = 2020-05-19 12:26:10
t_delta         = 0.0071
t_end           = 2020-05-19 12:26:10

[ior-hard-write]
t_start         = 2020-05-19 12:26:10
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/file -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 8500 -w -D 300 -a POSIX
throughput-stonewall = 13.22
score           = 0.111253
t_delta         = 401.6680
t_end           = 2020-05-19 12:32:52

[mdtest-hard-write]
t_start         = 2020-05-19 12:32:52
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall -C -W 300 -a POSIX
rate-stonewall  = 9.707868
score           = 9.683823
t_delta         = 310.3331
t_end           = 2020-05-19 12:38:02

[find]
t_start         = 2020-05-19 12:38:02
exe             = ./pfind /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app -newer /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/timestampfile -size 3901c -name *01* -C -H 1 -q 10000
found           = 108096
total-files     = 7394766
score           = 62.837501
t_delta         = 117.7972
t_end           = 2020-05-19 12:40:00

[ior-easy-read]
t_start         = 2020-05-19 12:40:00
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX
score           = 25.728265
t_delta         = 571.8632
t_end           = 2020-05-19 12:49:32

[mdtest-easy-stat]
t_start         = 2020-05-19 12:49:32
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy-stonewall -T -a POSIX
score           = 37.680302
t_delta         = 116.6023
t_end           = 2020-05-19 12:51:29

[ior-hard-read]
t_start         = 2020-05-19 12:51:29
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/file -O stoneWallingStatusFile=/mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 8500 -r -R -a POSIX
score           = 0.789003
t_delta         = 56.8667
t_end           = 2020-05-19 12:52:25

[mdtest-hard-stat]
t_start         = 2020-05-19 12:52:25
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall -T -a POSIX
score           = 29.886608
t_delta         = 100.5647
t_end           = 2020-05-19 12:54:06

[mdtest-easy-delete]
t_start         = 2020-05-19 12:54:06
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-easy-stonewall -r -a POSIX
score           = 7.066001
t_delta         = 621.9331
t_end           = 2020-05-19 13:04:28

[mdtest-hard-read]
t_start         = 2020-05-19 13:04:28
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 10.696823
t_delta         = 280.9237
t_end           = 2020-05-19 13:09:09

[mdtest-hard-delete]
t_start         = 2020-05-19 13:09:09
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard -x /mnt/lustre01/work/k20200/k202079/io500-data//2020.05.19-12.12.45-app/mdtest-hard-stonewall -r -a POSIX
score           = 8.049258
t_delta         = 373.9973
t_end           = 2020-05-19 13:15:23

[SCORE]
MD              = 16.570294
BW              = 2.887474
SCORE           = 6.917101 
hash            = 95DA6DD9
; END 2020-05-19 13:15:25
result_summary
IO500 version dev-17-gd9f8a8406c74
[RESULT]       ior-easy-write       30.780122 GiB/s  : time 478.156 seconds
[RESULT]    mdtest-easy-write       13.633290 kIOPS : time 322.338 seconds
[RESULT]       ior-hard-write        0.111253 GiB/s  : time 401.668 seconds
[RESULT]    mdtest-hard-write        9.683823 kIOPS : time 310.333 seconds
[RESULT]                 find       62.837501 kIOPS : time 117.797 seconds
[RESULT]        ior-easy-read       25.728265 GiB/s  : time 571.863 seconds
[RESULT]     mdtest-easy-stat       37.680302 kIOPS : time 116.602 seconds
[RESULT]        ior-hard-read        0.789003 GiB/s  : time 56.867 seconds
[RESULT]     mdtest-hard-stat       29.886608 kIOPS : time 100.565 seconds
[RESULT]   mdtest-easy-delete        7.066001 kIOPS : time 621.933 seconds
[RESULT]     mdtest-hard-read       10.696823 kIOPS : time 280.924 seconds
[RESULT]   mdtest-hard-delete        8.049258 kIOPS : time 373.997 seconds
[SCORE] Bandwidth 2.887474 GB/s : IOPS 16.570294 kiops : TOTAL 6.917101