BeeGFS on Oracle Cloud

Institution Oracle Cloud Infrastructure
Client Procs Per Node
Client Operating System Oracle Linux UEK
Client Operating System Version 7.7
Client Kernel Version 4.14.35-1902.6.6.el7uek.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 384 GB
Storage Interface NVMe
Network Ethernet
Software Version 7.1.5
OS Version OracleLinuxUEK77

INFORMATION

Client Nodes 270
Client Total Procs 3,240
Metadata Nodes 270
Metadata Storage Devices 1
Data Nodes 270
Data Storage Devices 1

METADATA

Easy Write 2,908.53 kIOP/s
Easy Stat 14,312.11 kIOP/s
Easy Delete 3,147.97 kIOP/s
Hard Write 2.22 kIOP/s
Hard Read 17.03 kIOP/s
Hard Stat 19.55 kIOP/s
Hard Delete 7.04 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 04:03:57 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.14-01.17.59-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-easy/stonewall -t 1m -b 200000m -F -r -R -a POSIX
Machine             : Linux inst-0aaux-brave-troll
TestID              : 0
StartTime           : Tue Jul 14 04:03:57 2020
Path                : /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app/ior-easy
FS                  : 1571.4 TiB   Used FS: 41.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.14-01.17.59-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 270
tasks               : 3240
clients per node    : 12
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 195.31 GiB
aggregate filesize  : 617.98 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 679477248000000.
WARNING: Stat() of aggregate file size      = 341050015088640.
WARNING: Using actual aggregate bytes moved = 341050015088640.
read      764657     765838     0.002123    204800000  1024.00    0.270898   424.70     0.387367   425.36     0   
Max Read:  764656.54 MiB/sec (801800.49 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       764656.54  764656.54  764656.54       0.00  764656.54  764656.54  764656.54       0.00  425.35521         NA            NA     0   3240  12    1   1     1        1         0    0      1 209715200000  1048576 325250624.0 POSIX      0
Finished            : Tue Jul 14 04:11:02 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 02:52:46 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.14-01.17.59-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-easy/stonewall -t 1m -b 200000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux inst-0aaux-brave-troll
TestID              : 0
StartTime           : Tue Jul 14 02:52:46 2020
Path                : /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app/ior-easy
FS                  : 1571.4 TiB   Used FS: 21.2%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.14-01.17.59-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 270
tasks               : 3240
clients per node    : 12
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 195.31 GiB
aggregate filesize  : 617.98 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 48079 max: 100386 -- min data: 47.0 GiB mean data: 83.3 GiB time: 300.1s
WARNING: Expected aggregate file size       = 679477248000000.
WARNING: Stat() of aggregate file size      = 341050015088640.
WARNING: Using actual aggregate bytes moved = 341050015088640.
WARNING: maybe caused by deadlineForStonewalling
write     495329     696807     0.001989    204800000  1024.00    189.46     466.77     0.405525   656.64     0   
Max Write: 495329.07 MiB/sec (519390.18 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      495329.07  495329.07  495329.07       0.00  495329.07  495329.07  495329.07       0.00  656.63547     300.09     920924.36     0   3240  12    1   1     1        1         0    0      1 209715200000  1048576 325250624.0 POSIX      0
Finished            : Tue Jul 14 03:05:56 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 04:13:35 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.14-01.17.59-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 2200000 -r -R -a POSIX
Machine             : Linux inst-0aaux-brave-troll
TestID              : 0
StartTime           : Tue Jul 14 04:13:35 2020
Path                : /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app/ior-hard
FS                  : 1571.4 TiB   Used FS: 41.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.14-01.17.59-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 2200000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 270
tasks               : 3240
clients per node    : 12
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 304.75 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 16793 max: 16793 -- min data: 0.7 GiB mean data: 0.7 GiB time: 223.4s
WARNING: Expected aggregate file size       = 335073024000000.
WARNING: Stat() of aggregate file size      = 2557673314560.
WARNING: Using actual aggregate bytes moved = 2557673314560.
read      10897      243397     223.54      45.91      45.91      0.222213   223.54     0.070848   223.83     0   
Max Read:  10897.38 MiB/sec (11426.73 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        10897.38   10897.38   10897.38       0.00  243080.64  243080.64  243080.64       0.00  223.83239     223.41      10918.12     0   3240  12    1   0     1        1         0    0 2200000    47008    47008 2439187.2 POSIX      0
Finished            : Tue Jul 14 04:17:19 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 03:18:48 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.14-01.17.59-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 2200000 -w -D 300 -a POSIX
Machine             : Linux inst-0aaux-brave-troll
TestID              : 0
StartTime           : Tue Jul 14 03:18:48 2020
Path                : /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app/ior-hard
FS                  : 1571.4 TiB   Used FS: 41.5%   Inodes: 0.0 Mi   Used Inodes: -nan%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : ./out//2020.07.14-01.17.59-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 2200000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 270
tasks               : 3240
clients per node    : 12
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 304.75 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1710 max: 16793 -- min data: 0.1 GiB mean data: 0.2 GiB time: 321.4s
WARNING: Expected aggregate file size       = 335073024000000.
WARNING: Stat() of aggregate file size      = 2557673314560.
WARNING: Using actual aggregate bytes moved = 2557673314560.
WARNING: maybe caused by deadlineForStonewalling
write     1349.26    30102      1722.66     45.91      45.91      0.274388   1807.51    0.013749   1807.79    0   
Max Write: 1349.26 MiB/sec (1414.80 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        1349.26    1349.26    1349.26       0.00   30097.07   30097.07   30097.07       0.00 1807.79484     321.40       1952.46     0   3240  12    1   0     1        1         0    0 2200000    47008    47008 2439187.2 POSIX      0
Finished            : Tue Jul 14 03:52:06 2020
mdtest-easy-delete
-- started at 07/14/2020 04:18:18 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '5000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './out//2020.07.14-01.17.59-app/mdtest-easy' '-x' './out//2020.07.14-01.17.59-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 41.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 16200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    3147976.309    3147971.802    3147974.377          0.697
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.027          0.027          0.027          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        695.611        695.610        695.610          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         36.528         36.528         36.528          0.000
-- finished at 07/14/2020 04:30:34 --

mdtest-easy-stat
-- started at 07/14/2020 04:11:02 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '5000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './out//2020.07.14-01.17.59-app/mdtest-easy' '-x' './out//2020.07.14-01.17.59-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 41.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 16200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :   14312128.656   14312038.566   14312099.437          5.506
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        153.002        153.001        153.001          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 04:13:35 --

mdtest-easy-write
-- started at 07/14/2020 03:05:56 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '5000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' './out//2020.07.14-01.17.59-app/mdtest-easy' '-x' './out//2020.07.14-01.17.59-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 40.9%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 16200000000 files
Continue stonewall hit min: 51730 max: 675853 avg: 316338.7 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    2908536.665    2908525.261    2908530.064          2.509
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    3623543.965             NA
   Tree creation             :          0.058          0.058          0.058          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        752.878        752.875        752.876          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        282.855             NA
   Tree creation             :         17.189         17.189         17.189          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 03:18:48 --

mdtest-hard-delete
-- started at 07/14/2020 04:31:42 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '160000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.14-01.17.59-app/mdtest-hard' '-x' './out//2020.07.14-01.17.59-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 41.1%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 518400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       7035.877       7035.847       7035.867          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.769          0.769          0.769          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        162.556        162.555        162.556          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.301          1.301          1.301          0.000
-- finished at 07/14/2020 04:34:25 --

mdtest-hard-read
-- started at 07/14/2020 04:30:34 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '160000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.14-01.17.59-app/mdtest-hard' '-x' './out//2020.07.14-01.17.59-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 41.1%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 518400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      17027.467      17027.353      17027.420          0.009
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         67.170         67.169         67.169          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 04:31:42 --

mdtest-hard-stat
-- started at 07/14/2020 04:17:19 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '160000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.14-01.17.59-app/mdtest-hard' '-x' './out//2020.07.14-01.17.59-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 41.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 518400000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      19554.766      19554.621      19554.675          0.012
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         58.488         58.488         58.488          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 04:18:18 --

mdtest-hard-write
-- started at 07/14/2020 03:52:06 --

mdtest-3.3.0+dev was launched with 3240 total task(s) on 270 node(s)
Command line used: ./mdtest '-n' '160000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' './out//2020.07.14-01.17.59-app/mdtest-hard' '-x' './out//2020.07.14-01.17.59-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /mnt/beeond/io500-app/out/2020.07.14-01.17.59-app
FS: 1571.4 TiB   Used FS: 41.7%   Inodes: 0.0 Mi   Used Inodes: -nan%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
3240 tasks, 518400000 files
Continue stonewall hit min: 238 max: 353 avg: 273.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2221.466       2221.464       2221.465          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2936.237             NA
   Tree creation             :        142.131        142.131        142.131          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        514.850        514.849        514.850          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.330             NA
   Tree creation             :          0.007          0.007          0.007          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 04:00:42 --

result
version         = io500-isc20_v4
config-hash     = 65073174
result-dir      = ./results//2020.07.14-01.17.59-app
; START 2020-07-14 02:52:46
; WARNING Using internal pfind, will ignore any arguments to the external script


[ior-easy-write]
t_start         = 2020-07-14 02:52:46
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.14-01.17.59-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-easy/stonewall -t 1m -b 200000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 1331690.67
score           = 483.719797
t_delta         = 789.8664
t_end           = 2020-07-14 03:05:56

[mdtest-easy-write]
t_start         = 2020-07-14 03:05:56
exe             = ./mdtest -n 5000000 -u -L -F -P -N 1 -d ./out//2020.07.14-01.17.59-app/mdtest-easy -x ./out//2020.07.14-01.17.59-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 3623.543965
score           = 2908.530413
t_delta         = 772.1078
t_end           = 2020-07-14 03:18:48

[timestamp]
t_start         = 2020-07-14 03:18:48
t_delta         = 0.0009
t_end           = 2020-07-14 03:18:48

[ior-hard-write]
t_start         = 2020-07-14 03:18:48
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.14-01.17.59-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 2200000 -w -D 300 -a POSIX
throughput-stonewall = 1098.30
score           = 1.317638
t_delta         = 1997.5676
t_end           = 2020-07-14 03:52:06

[mdtest-hard-write]
t_start         = 2020-07-14 03:52:06
exe             = ./mdtest -n 160000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.14-01.17.59-app/mdtest-hard -x ./out//2020.07.14-01.17.59-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 2.936237
score           = 2.221464
t_delta         = 515.9973
t_end           = 2020-07-14 04:00:42

[find]
t_start         = 2020-07-14 04:00:42
exe             = ./pfind ./out//2020.07.14-01.17.59-app -newer ./out//2020.07.14-01.17.59-app/timestampfile -size 3901c -name *01* -C -N -H 1 -q 15000
found           = 31420
total-files     = 2190910686
score           = 11217.629155
t_delta         = 195.3589
t_end           = 2020-07-14 04:03:57

[ior-easy-read]
t_start         = 2020-07-14 04:03:57
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o ./out//2020.07.14-01.17.59-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-easy/stonewall -t 1m -b 200000m -F -r -R -a POSIX
score           = 746.734899
t_delta         = 425.4484
t_end           = 2020-07-14 04:11:02

[mdtest-easy-stat]
t_start         = 2020-07-14 04:11:02
exe             = ./mdtest -n 5000000 -u -L -F -P -N 1 -d ./out//2020.07.14-01.17.59-app/mdtest-easy -x ./out//2020.07.14-01.17.59-app/mdtest-easy-stonewall -T -a POSIX
score           = 14312.108177
t_delta         = 153.0121
t_end           = 2020-07-14 04:13:35

[ior-hard-read]
t_start         = 2020-07-14 04:13:35
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o ./out//2020.07.14-01.17.59-app/ior-hard/file -O stoneWallingStatusFile=./out//2020.07.14-01.17.59-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 2200000 -r -R -a POSIX
score           = 10.641976
t_delta         = 223.9856
t_end           = 2020-07-14 04:17:19

[mdtest-hard-stat]
t_start         = 2020-07-14 04:17:19
exe             = ./mdtest -n 160000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.14-01.17.59-app/mdtest-hard -x ./out//2020.07.14-01.17.59-app/mdtest-hard-stonewall -T -a POSIX
score           = 19.554671
t_delta         = 58.5132
t_end           = 2020-07-14 04:18:18

[mdtest-easy-delete]
t_start         = 2020-07-14 04:18:18
exe             = ./mdtest -n 5000000 -u -L -F -P -N 1 -d ./out//2020.07.14-01.17.59-app/mdtest-easy -x ./out//2020.07.14-01.17.59-app/mdtest-easy-stonewall -r -a POSIX
score           = 3147.974543
t_delta         = 736.4367
t_end           = 2020-07-14 04:30:34

[mdtest-hard-read]
t_start         = 2020-07-14 04:30:34
exe             = ./mdtest -n 160000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.14-01.17.59-app/mdtest-hard -x ./out//2020.07.14-01.17.59-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 17.027393
t_delta         = 67.1973
t_end           = 2020-07-14 04:31:42

[mdtest-hard-delete]
t_start         = 2020-07-14 04:31:42
exe             = ./mdtest -n 160000 -t -w 3901 -e 3901 -P -N 1 -F -d ./out//2020.07.14-01.17.59-app/mdtest-hard -x ./out//2020.07.14-01.17.59-app/mdtest-hard-stonewall -r -a POSIX
score           = 7.035865
t_delta         = 163.8746
t_end           = 2020-07-14 04:34:25

[SCORE]
MD              = 229.328552
BW              = 47.439998
SCORE           = 104.304104 
hash            = 5F141BF5
; END 2020-07-14 04:34:25
result_summary
IO500 version io500-isc20_v4
[RESULT]       ior-easy-write      483.719797 GiB/s  : time 789.866 seconds
[RESULT]    mdtest-easy-write     2908.530413 kIOPS : time 772.108 seconds
[RESULT]       ior-hard-write        1.317638 GiB/s  : time 1997.568 seconds
[RESULT]    mdtest-hard-write        2.221464 kIOPS : time 515.997 seconds
[RESULT]                 find    11217.629155 kIOPS : time 195.359 seconds
[RESULT]        ior-easy-read      746.734899 GiB/s  : time 425.448 seconds
[RESULT]     mdtest-easy-stat    14312.108177 kIOPS : time 153.012 seconds
[RESULT]        ior-hard-read       10.641976 GiB/s  : time 223.986 seconds
[RESULT]     mdtest-hard-stat       19.554671 kIOPS : time 58.513 seconds
[RESULT]   mdtest-easy-delete     3147.974543 kIOPS : time 736.437 seconds
[RESULT]     mdtest-hard-read       17.027393 kIOPS : time 67.197 seconds
[RESULT]   mdtest-hard-delete        7.035865 kIOPS : time 163.875 seconds
[SCORE] Bandwidth 47.439998 GB/s : IOPS 229.328552 kiops : TOTAL 104.304104