SpectrumScale on Oracle Cloud

Institution Oracle Cloud Infrastructure
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8
Client Kernel Version 3.10.0-1127.10.1.el7.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 768 GB
Storage Interface iSCSI
Network Ethernet
Software Version 5.0.5.0
OS Version CentOS78

INFORMATION

Client Nodes 48
Client Total Procs 768
Metadata Nodes 40
Metadata Storage Devices 10
Data Nodes 40
Data Storage Devices 10

METADATA

Easy Write 986.28 kIOP/s
Easy Stat 796.96 kIOP/s
Easy Delete 408.90 kIOP/s
Hard Write 18.90 kIOP/s
Hard Read 28.14 kIOP/s
Hard Stat 48.68 kIOP/s
Hard Delete 10.08 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 16:26:19 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-easy/stonewall -t 2m -b 45056m -F -r -R -a POSIX
Machine             : Linux ss-compute-1
TestID              : 0
StartTime           : Thu Jul  9 16:26:19 2020
Path                : /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app/ior-easy
FS                  : 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 53.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datafiles/2020.07.09-15.51.08-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 768
clients per node    : 16
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 44 GiB
aggregate filesize  : 33 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
read      95701      47858      0.016047    46137344   2048.00    0.043558   361.52     0.013876   361.57     0   
Max Read:  95700.99 MiB/sec (100349.77 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        95700.99   95700.99   95700.99       0.00   47850.50   47850.50   47850.50       0.00  361.57418         NA            NA     0    768  16    1   1     1        1         0    0      1 47244640256  2097152 34603008.0 POSIX      0
Finished            : Thu Jul  9 16:32:21 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 15:51:17 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-easy/stonewall -t 2m -b 45056m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux ss-compute-1
TestID              : 0
StartTime           : Thu Jul  9 15:51:17 2020
Path                : /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app/ior-easy
FS                  : 390.6 TiB   Used FS: 0.5%   Inodes: 600.0 Mi   Used Inodes: 0.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datafiles/2020.07.09-15.51.08-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 768
clients per node    : 16
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 44 GiB
aggregate filesize  : 33 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 2478 max: 22528 -- min data: 4.8 GiB mean data: 38.4 GiB time: 320.7s
write     85339      42713      0.014584    46137344   2048.00    0.401613   405.06     0.014404   405.48     0   
Max Write: 85338.61 MiB/sec (89484.02 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       85338.61   85338.61   85338.61       0.00   42669.31   42669.31   42669.31       0.00  405.47891     320.73      94069.50     0    768  16    1   1     1        1         0    0      1 47244640256  2097152 34603008.0 POSIX      0
Finished            : Thu Jul  9 15:58:02 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 16:39:11 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-hard/file -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 95000 -r -R -a POSIX
Machine             : Linux ss-compute-1
TestID              : 0
StartTime           : Thu Jul  9 16:39:11 2020
Path                : /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app/ior-hard
FS                  : 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 53.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datafiles/2020.07.09-15.51.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 95000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 768
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 3.12 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 95000 max: 95000 -- min data: 4.2 GiB mean data: 4.2 GiB time: 79.2s
read      41257      921388     79.18       45.91      45.91      0.008029   79.18      0.090194   79.28      0   
Max Read:  41257.28 MiB/sec (43261.39 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        41257.28   41257.28   41257.28       0.00  920298.44  920298.44  920298.44       0.00   79.27863      79.17      41314.87     0    768  16    1   0     1        1         0    0  95000    47008    47008 3270820.2 POSIX      0
Finished            : Thu Jul  9 16:40:30 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 16:04:04 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-hard/file -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 95000 -w -D 300 -a POSIX
Machine             : Linux ss-compute-1
TestID              : 0
StartTime           : Thu Jul  9 16:04:04 2020
Path                : /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app/ior-hard
FS                  : 390.6 TiB   Used FS: 9.0%   Inodes: 600.0 Mi   Used Inodes: 51.9%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./datafiles/2020.07.09-15.51.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 95000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 48
tasks               : 768
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 3.12 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 24938 max: 95000 -- min data: 1.1 GiB mean data: 2.1 GiB time: 300.1s
write     4844       108071     300.82      45.91      45.91      0.015307   675.11     0.057062   675.18     0   
Max Write: 4844.35 MiB/sec (5079.67 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        4844.35    4844.35    4844.35       0.00  108059.69  108059.69  108059.69       0.00  675.18242     300.09       5524.61     0    768  16    1   0     1        1         0    0  95000    47008    47008 3270820.2 POSIX      0
Finished            : Thu Jul  9 16:15:19 2020
mdtest-easy-delete
-- started at 07/09/2020 16:43:39 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-easy' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 53.4%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 768000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     408899.217     408897.316     408898.339          0.374
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.066          0.066          0.066          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        799.277        799.274        799.275          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         15.148         15.148         15.148          0.000
-- finished at 07/09/2020 16:57:13 --

mdtest-easy-stat
-- started at 07/09/2020 16:32:21 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-easy' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 53.4%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 768000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     796968.513     796960.754     796965.467          1.271
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        410.086        410.082        410.084          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 16:39:11 --

mdtest-easy-write
-- started at 07/09/2020 15:58:02 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-easy' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.0%   Inodes: 600.0 Mi   Used Inodes: 0.0%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 768000000 files
Continue stonewall hit min: 370151 max: 425550 avg: 396789.7 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     986281.021     986278.243     986279.451          0.501
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    1046249.265             NA
   Tree creation             :          0.114          0.114          0.114          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        331.369        331.368        331.369          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        291.264             NA
   Tree creation             :          8.757          8.757          8.757          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 16:04:04 --

mdtest-hard-delete
-- started at 07/09/2020 17:02:40 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '30000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-hard' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 1.5%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 23040000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      10078.185      10078.141      10078.159          0.008
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         18.821         18.821         18.821          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        910.873        910.869        910.871          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.053          0.053          0.053          0.000
-- finished at 07/09/2020 17:17:51 --

mdtest-hard-read
-- started at 07/09/2020 16:57:13 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '30000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-hard' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 1.6%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 23040000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      28137.867      28137.516      28137.696          0.063
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        326.251        326.247        326.249          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 17:02:40 --

mdtest-hard-stat
-- started at 07/09/2020 16:40:30 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '30000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-hard' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 53.4%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 23040000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      48676.495      48675.569      48675.870          0.122
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        188.594        188.590        188.593          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 16:43:39 --

mdtest-hard-write
-- started at 07/09/2020 16:15:19 --

mdtest-3.3.0+dev was launched with 768 total task(s) on 48 node(s)
Command line used: ./mdtest '-n' '30000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './datafiles/2020.07.09-15.51.08-app/mdtest-hard' '-x' './datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /gpfs/fs1/io500-app-master/datafiles/2020.07.09-15.51.08-app
FS: 390.6 TiB   Used FS: 9.8%   Inodes: 600.0 Mi   Used Inodes: 51.9%

Nodemap: 111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
768 tasks, 23040000 files
Continue stonewall hit min: 8408 max: 11953 avg: 9168.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      18901.388      18901.350      18901.367          0.006
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      23458.947             NA
   Tree creation             :       2885.403       2885.403       2885.403          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        485.675        485.674        485.674          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.155             NA
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 16:23:47 --

result
version         = 
config-hash     = 6237D43B
result-dir      = ./results/2020.07.09-15.51.08-app
; START 2020-07-09 15:51:17


[ior-easy-write]
t_start         = 2020-07-09 15:51:17
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-easy/stonewall -t 2m -b 45056m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 55806.29
score           = 83.338489
t_delta         = 405.5145
t_end           = 2020-07-09 15:58:02

[mdtest-easy-write]
t_start         = 2020-07-09 15:58:02
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datafiles/2020.07.09-15.51.08-app/mdtest-easy -x ./datafiles/2020.07.09-15.51.08-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 1046.249265
score           = 986.278541
t_delta         = 361.5437
t_end           = 2020-07-09 16:04:04

[timestamp]
t_start         = 2020-07-09 16:04:04
t_delta         = 0.0015
t_end           = 2020-07-09 16:04:04

[ior-hard-write]
t_start         = 2020-07-09 16:04:04
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-hard/file -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 95000 -w -D 300 -a POSIX
throughput-stonewall = 1841.59
score           = 4.730811
t_delta         = 675.2042
t_end           = 2020-07-09 16:15:19

[mdtest-hard-write]
t_start         = 2020-07-09 16:15:19
exe             = ./mdtest -n 30000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datafiles/2020.07.09-15.51.08-app/mdtest-hard -x ./datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 23.458947
score           = 18.901374
t_delta         = 508.1534
t_end           = 2020-07-09 16:23:47

[find]
t_start         = 2020-07-09 16:23:47
exe             = ./pfind ./datafiles/2020.07.09-15.51.08-app -newer ./datafiles/2020.07.09-15.51.08-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 333279
total-files     = 336003078
score           = 2211.690493
t_delta         = 151.9541
t_end           = 2020-07-09 16:26:19

[ior-easy-read]
t_start         = 2020-07-09 16:26:19
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-easy/stonewall -t 2m -b 45056m -F -r -R -a POSIX
score           = 93.458002
t_delta         = 361.6672
t_end           = 2020-07-09 16:32:21

[mdtest-easy-stat]
t_start         = 2020-07-09 16:32:21
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datafiles/2020.07.09-15.51.08-app/mdtest-easy -x ./datafiles/2020.07.09-15.51.08-app/mdtest-easy-stonewall -T -a POSIX
score           = 796.962313
t_delta         = 410.1876
t_end           = 2020-07-09 16:39:11

[ior-hard-read]
t_start         = 2020-07-09 16:39:11
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./datafiles/2020.07.09-15.51.08-app/ior-hard/file -O stoneWallingStatusFile=./datafiles/2020.07.09-15.51.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 95000 -r -R -a POSIX
score           = 40.290308
t_delta         = 79.3731
t_end           = 2020-07-09 16:40:30

[mdtest-hard-stat]
t_start         = 2020-07-09 16:40:30
exe             = ./mdtest -n 30000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datafiles/2020.07.09-15.51.08-app/mdtest-hard -x ./datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall -T -a POSIX
score           = 48.675680
t_delta         = 188.6983
t_end           = 2020-07-09 16:43:39

[mdtest-easy-delete]
t_start         = 2020-07-09 16:43:39
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d ./datafiles/2020.07.09-15.51.08-app/mdtest-easy -x ./datafiles/2020.07.09-15.51.08-app/mdtest-easy-stonewall -r -a POSIX
score           = 408.897770
t_delta         = 814.5421
t_end           = 2020-07-09 16:57:13

[mdtest-hard-read]
t_start         = 2020-07-09 16:57:13
exe             = ./mdtest -n 30000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datafiles/2020.07.09-15.51.08-app/mdtest-hard -x ./datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 28.137630
t_delta         = 326.3548
t_end           = 2020-07-09 17:02:40

[mdtest-hard-delete]
t_start         = 2020-07-09 17:02:40
exe             = ./mdtest -n 30000 -t -w 3901 -e 3901 -P -N 1 -F -d ./datafiles/2020.07.09-15.51.08-app/mdtest-hard -x ./datafiles/2020.07.09-15.51.08-app/mdtest-hard-stonewall -r -a POSIX
score           = 10.078145
t_delta         = 911.0303
t_end           = 2020-07-09 17:17:51

[SCORE]
MD              = 144.055965
BW              = 34.905959
SCORE           = 70.911294 
hash            = 8112D84
; END 2020-07-09 17:17:52
result_summary
IO500 version 
[RESULT]       ior-easy-write       83.338489 GiB/s  : time 405.515 seconds
[RESULT]    mdtest-easy-write      986.278541 kIOPS : time 361.544 seconds
[RESULT]       ior-hard-write        4.730811 GiB/s  : time 675.204 seconds
[RESULT]    mdtest-hard-write       18.901374 kIOPS : time 508.153 seconds
[RESULT]                 find     2211.690493 kIOPS : time 151.954 seconds
[RESULT]        ior-easy-read       93.458002 GiB/s  : time 361.667 seconds
[RESULT]     mdtest-easy-stat      796.962313 kIOPS : time 410.188 seconds
[RESULT]        ior-hard-read       40.290308 GiB/s  : time 79.373 seconds
[RESULT]     mdtest-hard-stat       48.675680 kIOPS : time 188.698 seconds
[RESULT]   mdtest-easy-delete      408.897770 kIOPS : time 814.542 seconds
[RESULT]     mdtest-hard-read       28.137630 kIOPS : time 326.355 seconds
[RESULT]   mdtest-hard-delete       10.078145 kIOPS : time 911.030 seconds
[SCORE] Bandwidth 34.905959 GB/s : IOPS 144.055965 kiops : TOTAL 70.911294