BeeGFS on Oracle Cloud

Institution Oracle Cloud Infrastructure
Client Procs Per Node
Client Operating System Oracle Linux UEK
Client Operating System Version 7.7
Client Kernel Version 4.14.35-1902.6.6.el7uek.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 384 GB
Storage Interface NVMe
Network Ethernet
Software Version 7.1.5
OS Version OracleLinuxUEK77

INFORMATION

Client Nodes 10
Client Total Procs 160
Metadata Nodes 10
Metadata Storage Devices 1
Data Nodes 10
Data Storage Devices 10

METADATA

Easy Write 175.09 kIOP/s
Easy Stat 685.32 kIOP/s
Easy Delete 188.63 kIOP/s
Hard Write 3.02 kIOP/s
Hard Read 19.88 kIOP/s
Hard Stat 64.08 kIOP/s
Hard Delete 8.31 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 12:55:45 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.09-12.23.20-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-easy/stonewall -t 1m -b 3600000m -F -r -R -a POSIX
Machine             : Linux inst-052fy-united-hermit
TestID              : 0
StartTime           : Thu Jul  9 12:55:45 2020
Path                : /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app/ior-easy
FS                  : 58.2 TiB   Used FS: 44.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.09-12.23.20-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 160
clients per node    : 16
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 3.43 TiB
aggregate filesize  : 549.32 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 603979776000000.
WARNING: Stat() of aggregate file size      = 12689615093760.
WARNING: Using actual aggregate bytes moved = 12689615093760.
read      29768      29773      0.000113    3686400000 1024.00    0.008738   406.47     0.061920   406.54     0   
Max Read:  29767.66 MiB/sec (31213.65 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        29767.66   29767.66   29767.66       0.00   29767.66   29767.66   29767.66       0.00  406.54052         NA            NA     0    160  16    1   1     1        1         0    0      1 3774873600000  1048576 12101760.0 POSIX      0
Finished            : Thu Jul  9 13:02:32 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 12:23:27 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.09-12.23.20-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-easy/stonewall -t 1m -b 3600000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux inst-052fy-united-hermit
TestID              : 0
StartTime           : Thu Jul  9 12:23:27 2020
Path                : /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app/ior-easy
FS                  : 58.2 TiB   Used FS: 22.9%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.09-12.23.20-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 160
clients per node    : 16
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 3.43 TiB
aggregate filesize  : 549.32 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 58455 max: 75636 -- min data: 57.1 GiB mean data: 63.1 GiB time: 300.1s
WARNING: Expected aggregate file size       = 603979776000000.
WARNING: Stat() of aggregate file size      = 12689615093760.
WARNING: Using actual aggregate bytes moved = 12689615093760.
WARNING: maybe caused by deadlineForStonewalling
write     26559      26566      0.000121    3686400000 1024.00    0.017859   455.54     0.096679   455.65     0   
Max Write: 26559.11 MiB/sec (27849.24 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       26559.11   26559.11   26559.11       0.00   26559.11   26559.11   26559.11       0.00  455.65384     300.07      34438.99     0    160  16    1   1     1        1         0    0      1 3774873600000  1048576 12101760.0 POSIX      0
Finished            : Thu Jul  9 12:31:03 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 13:04:51 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.09-12.23.20-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 40000000 -r -R -a POSIX
Machine             : Linux inst-052fy-united-hermit
TestID              : 0
StartTime           : Thu Jul  9 13:04:51 2020
Path                : /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app/ior-hard
FS                  : 58.2 TiB   Used FS: 44.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.09-12.23.20-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 40000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 160
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 273.62 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 107323 max: 107323 -- min data: 4.7 GiB mean data: 4.7 GiB time: 72.5s
WARNING: Expected aggregate file size       = 300851200000000.
WARNING: Stat() of aggregate file size      = 807206333440.
WARNING: Using actual aggregate bytes moved = 807206333440.
read      10189      227386     75.44       45.91      45.91      0.032788   75.52      0.006741   75.56      0   
Max Read:  10188.51 MiB/sec (10683.43 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        10188.51   10188.51   10188.51       0.00  227268.29  227268.29  227268.29       0.00   75.55687      72.46      10623.87     0    160  16    1   0     1        1         0    0 40000000    47008    47008  769811.9 POSIX      0
Finished            : Thu Jul  9 13:06:07 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 12:40:10 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.09-12.23.20-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 40000000 -w -D 300 -a POSIX
Machine             : Linux inst-052fy-united-hermit
TestID              : 0
StartTime           : Thu Jul  9 12:40:10 2020
Path                : /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app/ior-hard
FS                  : 58.2 TiB   Used FS: 43.4%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.09-12.23.20-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 40000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 160
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 273.62 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 48410 max: 107323 -- min data: 2.1 GiB mean data: 3.3 GiB time: 303.0s
WARNING: Expected aggregate file size       = 300851200000000.
WARNING: Stat() of aggregate file size      = 807206333440.
WARNING: Using actual aggregate bytes moved = 807206333440.
WARNING: maybe caused by deadlineForStonewalling
write     1583.66    35327      480.78      45.91      45.91      0.012995   486.08     0.001646   486.10     0   
Max Write: 1583.66 MiB/sec (1660.59 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        1583.66    1583.66    1583.66       0.00   35325.63   35325.63   35325.63       0.00  486.09690     303.01       1808.12     0    160  16    1   0     1        1         0    0 40000000    47008    47008  769811.9 POSIX      0
Finished            : Thu Jul  9 12:48:17 2020
mdtest-easy-delete
-- started at 07/09/2020 13:06:24 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '50000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './out//2020.07.09-12.23.20-app/mdtest-easy' '-x' './out//2020.07.09-12.23.20-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 44.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 8000000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     188625.425     188625.252     188625.324          0.048
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.038          0.038          0.038          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        507.837        507.837        507.837          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         26.418         26.418         26.418          0.000
-- finished at 07/09/2020 13:15:22 --

mdtest-easy-stat
-- started at 07/09/2020 13:02:32 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '50000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './out//2020.07.09-12.23.20-app/mdtest-easy' '-x' './out//2020.07.09-12.23.20-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 44.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 8000000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     685321.453     685320.663     685321.109          0.184
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        139.775        139.775        139.775          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 13:04:51 --

mdtest-easy-write
-- started at 07/09/2020 12:31:03 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '50000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './out//2020.07.09-12.23.20-app/mdtest-easy' '-x' './out//2020.07.09-12.23.20-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 42.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 8000000000 files
Continue stonewall hit min: 345667 max: 598693 avg: 419519.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     175091.103     175090.955     175091.035          0.041
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     223787.400             NA
   Tree creation             :         17.271         17.271         17.271          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        547.092        547.092        547.092          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.941             NA
   Tree creation             :          0.058          0.058          0.058          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 12:40:10 --

mdtest-hard-delete
-- started at 07/09/2020 13:16:17 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '20000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.09-12.23.20-app/mdtest-hard' '-x' './out//2020.07.09-12.23.20-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 44.0%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 3200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       8306.196       8306.179       8306.188          0.005
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.731          0.731          0.731          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        131.988        131.988        131.988          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.367          1.367          1.367          0.000
-- finished at 07/09/2020 13:18:30 --

mdtest-hard-read
-- started at 07/09/2020 13:15:22 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '20000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.09-12.23.20-app/mdtest-hard' '-x' './out//2020.07.09-12.23.20-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 44.0%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 3200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      19883.770      19883.724      19883.743          0.011
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         55.137         55.136         55.136          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 13:16:17 --

mdtest-hard-stat
-- started at 07/09/2020 13:06:07 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '20000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.09-12.23.20-app/mdtest-hard' '-x' './out//2020.07.09-12.23.20-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 44.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 3200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      64077.080      64076.871      64077.000          0.044
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         17.109         17.109         17.109          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 13:06:24 --

mdtest-hard-write
-- started at 07/09/2020 12:48:17 --

mdtest-3.3.0+dev was launched with 160 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '20000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.09-12.23.20-app/mdtest-hard' '-x' './out//2020.07.09-12.23.20-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.09-12.23.20-app
FS: 58.2 TiB   Used FS: 44.6%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 1111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
160 tasks, 3200000000 files
Continue stonewall hit min: 5877 max: 6852 avg: 6125.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       3019.698       3019.695       3019.696          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       3266.328             NA
   Tree creation             :        670.552        670.552        670.552          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        363.057        363.056        363.056          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.072             NA
   Tree creation             :          0.001          0.001          0.001          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 12:54:20 --

result
version         = io500-isc20_v4
config-hash     = 6EC5039C
result-dir      = ./results//2020.07.09-12.23.20-app
; START 2020-07-09 12:23:27
; WARNING Using internal pfind, will ignore any arguments to the external script


[ior-easy-write]
t_start         = 2020-07-09 12:23:27
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.09-12.23.20-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-easy/stonewall -t 1m -b 3600000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 3543.73
score           = 25.936630
t_delta         = 456.0141
t_end           = 2020-07-09 12:31:03

[mdtest-easy-write]
t_start         = 2020-07-09 12:31:03
exe             = ./mdtest -n 50000000 -u -L -F -P -N 1 -d ./out//2020.07.09-12.23.20-app/mdtest-easy -x ./out//2020.07.09-12.23.20-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 223.787400
score           = 175.091016
t_delta         = 547.2886
t_end           = 2020-07-09 12:40:10

[timestamp]
t_start         = 2020-07-09 12:40:10
t_delta         = 0.0010
t_end           = 2020-07-09 12:40:10

[ior-hard-write]
t_start         = 2020-07-09 12:40:10
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.09-12.23.20-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 40000000 -w -D 300 -a POSIX
throughput-stonewall = 176.11
score           = 1.546542
t_delta         = 486.4584
t_end           = 2020-07-09 12:48:17

[mdtest-hard-write]
t_start         = 2020-07-09 12:48:17
exe             = ./mdtest -n 20000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.09-12.23.20-app/mdtest-hard -x ./out//2020.07.09-12.23.20-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 3.266328
score           = 3.019698
t_delta         = 363.1702
t_end           = 2020-07-09 12:54:20

[find]
t_start         = 2020-07-09 12:54:20
exe             = ./pfind ./out//2020.07.09-12.23.20-app -newer ./out//2020.07.09-12.23.20-app/timestampfile -size 3901c -name *01* -C -N -H 1 -q 15000
found           = 27204
total-files     = 96887366
score           = 1139.151517
t_delta         = 85.0810
t_end           = 2020-07-09 12:55:45

[ior-easy-read]
t_start         = 2020-07-09 12:55:45
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.09-12.23.20-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-easy/stonewall -t 1m -b 3600000m -F -r -R -a POSIX
score           = 29.069981
t_delta         = 406.5460
t_end           = 2020-07-09 13:02:32

[mdtest-easy-stat]
t_start         = 2020-07-09 13:02:32
exe             = ./mdtest -n 50000000 -u -L -F -P -N 1 -d ./out//2020.07.09-12.23.20-app/mdtest-easy -x ./out//2020.07.09-12.23.20-app/mdtest-easy-stonewall -T -a POSIX
score           = 685.320910
t_delta         = 139.7825
t_end           = 2020-07-09 13:04:51

[ior-hard-read]
t_start         = 2020-07-09 13:04:51
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.09-12.23.20-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.09-12.23.20-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 40000000 -r -R -a POSIX
score           = 9.949717
t_delta         = 75.5810
t_end           = 2020-07-09 13:06:07

[mdtest-hard-stat]
t_start         = 2020-07-09 13:06:07
exe             = ./mdtest -n 20000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.09-12.23.20-app/mdtest-hard -x ./out//2020.07.09-12.23.20-app/mdtest-hard-stonewall -T -a POSIX
score           = 64.076995
t_delta         = 17.1139
t_end           = 2020-07-09 13:06:24

[mdtest-easy-delete]
t_start         = 2020-07-09 13:06:24
exe             = ./mdtest -n 50000000 -u -L -F -P -N 1 -d ./out//2020.07.09-12.23.20-app/mdtest-easy -x ./out//2020.07.09-12.23.20-app/mdtest-easy-stonewall -r -a POSIX
score           = 188.625264
t_delta         = 537.3986
t_end           = 2020-07-09 13:15:22

[mdtest-hard-read]
t_start         = 2020-07-09 13:15:22
exe             = ./mdtest -n 20000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.09-12.23.20-app/mdtest-hard -x ./out//2020.07.09-12.23.20-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 19.883764
t_delta         = 55.4972
t_end           = 2020-07-09 13:16:17

[mdtest-hard-delete]
t_start         = 2020-07-09 13:16:17
exe             = ./mdtest -n 20000000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.09-12.23.20-app/mdtest-hard -x ./out//2020.07.09-12.23.20-app/mdtest-hard-stonewall -r -a POSIX
score           = 8.306189
t_delta         = 133.3605
t_end           = 2020-07-09 13:18:30

[SCORE]
MD              = 73.196137
BW              = 10.378455
SCORE           = 27.561982 
hash            = 86BFC27B
; END 2020-07-09 13:18:31
result_summary
IO500 version io500-isc20_v4
[RESULT]       ior-easy-write       25.936630 GiB/s  : time 456.014 seconds
[RESULT]    mdtest-easy-write      175.091016 kIOPS : time 547.289 seconds
[RESULT]       ior-hard-write        1.546542 GiB/s  : time 486.458 seconds
[RESULT]    mdtest-hard-write        3.019698 kIOPS : time 363.170 seconds
[RESULT]                 find     1139.151517 kIOPS : time 85.081 seconds
[RESULT]        ior-easy-read       29.069981 GiB/s  : time 406.546 seconds
[RESULT]     mdtest-easy-stat      685.320910 kIOPS : time 139.783 seconds
[RESULT]        ior-hard-read        9.949717 GiB/s  : time 75.581 seconds
[RESULT]     mdtest-hard-stat       64.076995 kIOPS : time 17.114 seconds
[RESULT]   mdtest-easy-delete      188.625264 kIOPS : time 537.399 seconds
[RESULT]     mdtest-hard-read       19.883764 kIOPS : time 55.497 seconds
[RESULT]   mdtest-hard-delete        8.306189 kIOPS : time 133.361 seconds
[SCORE] Bandwidth 10.378455 GB/s : IOPS 73.196137 kiops : TOTAL 27.561982