HAL

Institution National Center for Supercomputing Applications
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7
Client Kernel Version 4.18.0-147.0.3.el7.ppc64le

DATA SERVER

Storage Type NVMe
Volatile Memory 32
Storage Interface Infiniband
Network Infiniband
Software Version 5.0.5.0
OS Version 7.6

INFORMATION

Client Nodes 10
Client Total Procs 100
Metadata Nodes 8
Metadata Storage Devices 2
Data Nodes 8
Data Storage Devices 2

METADATA

Easy Write 74.67 kIOP/s
Easy Stat 107.80 kIOP/s
Easy Delete 32.59 kIOP/s
Hard Write 1.74 kIOP/s
Hard Read 51.80 kIOP/s
Hard Stat 106.70 kIOP/s
Hard Delete 0.47 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jun 27 07:41:29 2020
Command line        : ./ior -v -C -Q 1 -g -G 271 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
Machine             : Linux hal01
Start time skew across all tasks: 0.00 sec
TestID              : 0
StartTime           : Sat Jun 27 07:41:29 2020
Path                : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy
FS                  : 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 96.0%
Participating tasks: 100
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 100
clients per node    : 10
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 946.04 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Sat Jun 27 07:41:29 2020

WARNING: Expected aggregate file size       = 1040187392000000.
WARNING: Stat() of aggregate file size      = 5984013516800.
WARNING: Using actual aggregate bytes moved = 5984013516800.
read      13043      3261.24    0.000176    10158080000 4096       0.052272   437.47     0.010134   437.53     0   
Max Read:  13043.10 MiB/sec (13676.68 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        13043.10   13043.10   13043.10       0.00    3260.77    3260.77    3260.77       0.00  437.53412         NA            NA     0    100  10    1   1     1        1         0    0      1 10401873920000  4194304 5706800.0 POSIX      0
Finished            : Sat Jun 27 07:48:46 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jun 27 06:53:46 2020
Command line        : ./ior -v -C -Q 1 -g -G 271 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a posix
Machine             : Linux hal01
Start time skew across all tasks: 0.00 sec
TestID              : 0
StartTime           : Sat Jun 27 06:53:46 2020
Path                : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy
FS                  : 223.2 TiB   Used FS: 49.6%   Inodes: 194.5 Mi   Used Inodes: 81.4%
Participating tasks: 100
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : posix
apiVersion          : 
test filename       : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 100
clients per node    : 10
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 946.04 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Sat Jun 27 06:53:46 2020
0: stonewalling pairs accessed: 13343
stonewalling pairs accessed min: 9636 max: 14267 -- min data: 37.6 GiB mean data: 43.1 GiB time: 302.1s
WARNING: Expected aggregate file size       = 1040187392000000.
WARNING: Stat() of aggregate file size      = 5984013516800.
WARNING: Using actual aggregate bytes moved = 5984013516800.
WARNING: maybe caused by deadlineForStonewalling
write     14582      3648       0.000130    10158080000 4096       0.222788   391.12     0.009839   391.36     0   
Max Write: 14582.13 MiB/sec (15290.47 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       14582.13   14582.13   14582.13       0.00    3645.53    3645.53    3645.53       0.00  391.35571     302.11      14602.25     0    100  10    1   1     1        1         0    0      1 10401873920000  4194304 5706800.0 posix      0
Finished            : Sat Jun 27 07:00:18 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jun 27 07:52:57 2020
Command line        : ./ior -v -C -Q 1 -g -G 27 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/file -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
Machine             : Linux hal01
Start time skew across all tasks: 0.00 sec
TestID              : 0
StartTime           : Sat Jun 27 07:52:57 2020
Path                : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard
FS                  : 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 96.0%
Participating tasks: 100
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 100
clients per node    : 10
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 42.75 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Sat Jun 27 07:52:57 2020

0: stonewalling pairs accessed: 38731
stonewalling pairs accessed min: 38731 max: 38731 -- min data: 1.7 GiB mean data: 1.7 GiB time: 85.4s
WARNING: Expected aggregate file size       = 47008000000000.
WARNING: Stat() of aggregate file size      = 182066684800.
WARNING: Using actual aggregate bytes moved = 182066684800.
read      2032.41    45355      85.39       45.91      45.91      0.028343   85.39      0.009188   85.43      0   
Max Read:  2032.41 MiB/sec (2131.14 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         2032.41    2032.41    2032.41       0.00   45335.65   45335.65   45335.65       0.00   85.43167      85.36       2034.23     0    100  10    1   0     1        1         0    0 10000000    47008    47008  173632.3 POSIX      0
Finished            : Sat Jun 27 07:54:23 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jun 27 07:06:23 2020
Command line        : ./ior -v -C -Q 1 -g -G 27 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/file -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
Machine             : Linux hal01
Start time skew across all tasks: 0.00 sec
TestID              : 0
StartTime           : Sat Jun 27 07:06:23 2020
Path                : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard
FS                  : 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 94.5%
Participating tasks: 100
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 100
clients per node    : 10
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 42.75 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Sat Jun 27 07:06:24 2020
0: stonewalling pairs accessed: 38692
stonewalling pairs accessed min: 23456 max: 38731 -- min data: 1.0 GiB mean data: 1.2 GiB time: 300.1s
WARNING: Expected aggregate file size       = 47008000000000.
WARNING: Stat() of aggregate file size      = 182066684800.
WARNING: Using actual aggregate bytes moved = 182066684800.
WARNING: maybe caused by deadlineForStonewalling
write     491.27     10960      300.39      45.91      45.91      0.048423   353.37     0.015324   353.43     0   
Max Write: 491.27 MiB/sec (515.14 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write         491.27     491.27     491.27       0.00   10958.51   10958.51   10958.51       0.00  353.43318     300.06        420.29     0    100  10    1   0     1        1         0    0 10000000    47008    47008  173632.3 POSIX      0
Finished            : Sat Jun 27 07:12:17 2020
mdtest-easy-delete
-- started at 06/27/2020 07:54:51 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '2300000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 96.0%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 230000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      32592.657      32592.633      32592.650          0.008
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.220          0.220          0.220          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        831.019        831.018        831.019          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          4.544          4.544          4.544          0.000
-- finished at 06/27/2020 08:08:47 --

mdtest-easy-stat
-- started at 06/27/2020 07:48:46 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '2300000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 96.0%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 230000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     107798.987     107798.873     107798.951          0.037
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        251.256        251.256        251.256          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/27/2020 07:52:57 --

mdtest-easy-write
-- started at 06/27/2020 07:00:18 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '2300000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 51.5%   Inodes: 194.5 Mi   Used Inodes: 81.4%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 230000000 files
Continue stonewall hit min: 212325 max: 270851 avg: 245785.0 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      74672.356      74672.273      74672.330          0.024
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      82655.171             NA
   Tree creation             :          0.375          0.375          0.375          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        362.720        362.719        362.719          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        297.362             NA
   Tree creation             :          2.665          2.665          2.665          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/27/2020 07:06:23 --

mdtest-hard-delete
-- started at 06/27/2020 08:09:45 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '60000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 82.9%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 6000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        472.431        472.431        472.431          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         13.253         13.253         13.253          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       6378.709       6378.708       6378.709          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.075          0.075          0.075          0.000
-- finished at 06/27/2020 09:56:04 --

mdtest-hard-read
-- started at 06/27/2020 08:08:47 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '60000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 83.5%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 6000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      51803.698      51803.630      51803.664          0.018
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         58.172         58.172         58.172          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/27/2020 08:09:45 --

mdtest-hard-stat
-- started at 06/27/2020 07:54:23 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '60000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 96.0%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 6000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     106697.076     106696.941     106697.044          0.035
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         28.244         28.244         28.244          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/27/2020 07:54:51 --

mdtest-hard-write
-- started at 06/27/2020 07:12:17 --

mdtest-3.3.0+dev was launched with 100 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '60000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard' '-x' '/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app
FS: 223.2 TiB   Used FS: 52.1%   Inodes: 194.5 Mi   Used Inodes: 94.5%

Nodemap: 1111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 10 for each phase.
100 tasks, 6000000 files
Continue stonewall hit min: 3592 max: 30135 avg: 8038.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       1739.822       1739.821       1739.822          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2676.867             NA
   Tree creation             :       3886.982       3886.982       3886.982          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       1732.075       1732.074       1732.074          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.288             NA
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/27/2020 07:41:09 --

result
version         = io500-isc20-5-gc155836d31c1
config-hash     = 4000A0DA
result-dir      = ./results/2020.06.27-06.53.38-app
; START 2020-06-27 06:53:46
; WARNING Using internal pfind, will ignore any arguments to the external script


[ior-easy-write]
t_start         = 2020-06-27 06:53:46
exe             = ./ior -v -C -Q 1 -g -G 271 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a posix
result-file     = ./results/2020.06.27-06.53.38-app/ior-easy-write.txt
accessed-pairs  = 14267
throughput-stonewall = 1100.82
score           = 14.240362
t_delta         = 391.5540
t_end           = 2020-06-27 07:00:18

[mdtest-easy-write]
t_start         = 2020-06-27 07:00:18
exe             = ./mdtest -n 2300000 -u -L -F -P -N 1 -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-easy-write.txt
time            = 362.719476
rate-stonewall  = 82.655171
score           = 74.672307
t_delta         = 365.5163
t_end           = 2020-06-27 07:06:23

[timestamp]
t_start         = 2020-06-27 07:06:23
timestamp-file  = /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/timestampfile
t_delta         = 0.2809
t_end           = 2020-06-27 07:06:23

[ior-hard-write]
t_start         = 2020-06-27 07:06:23
exe             = ./ior -v -C -Q 1 -g -G 27 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/file -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/ior-hard-write.txt
accessed-pairs  = 38731
throughput-stonewall = 34.85
score           = 0.479759
t_delta         = 353.5270
t_end           = 2020-06-27 07:12:17

[mdtest-hard-write]
t_start         = 2020-06-27 07:12:17
exe             = ./mdtest -n 60000 -t -w 3901 -e 3901 -P -N 1 -F -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-hard-write.txt
time            = 1732.074194
rate-stonewall  = 2.676867
score           = 1.739822
t_delta         = 1732.2453
t_end           = 2020-06-27 07:41:09

[find]
t_start         = 2020-06-27 07:41:09
exe             = ./pfind /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app -newer /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/timestampfile -size 3901c -name *01* -C -q 10000
nproc           = 100
found           = 83300
total-files     = 30098706
score           = 1553.046946
t_delta         = 19.3941
t_end           = 2020-06-27 07:41:29

[ior-easy-read]
t_start         = 2020-06-27 07:41:29
exe             = ./ior -v -C -Q 1 -g -G 271 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/ior-easy-read.txt
score           = 12.737400
t_delta         = 437.5453
t_end           = 2020-06-27 07:48:46

[mdtest-easy-stat]
t_start         = 2020-06-27 07:48:46
exe             = ./mdtest -n 2300000 -u -L -F -P -N 1 -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy-stonewall -T -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-easy-stat.txt
time            = 251.255627
score           = 107.798979
t_delta         = 251.2753
t_end           = 2020-06-27 07:52:57

[ior-hard-read]
t_start         = 2020-06-27 07:52:57
exe             = ./ior -v -C -Q 1 -g -G 27 -k -e -o /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/file -O stoneWallingStatusFile=/gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/ior-hard-read.txt
score           = 1.984777
t_delta         = 85.4561
t_end           = 2020-06-27 07:54:23

[mdtest-hard-stat]
t_start         = 2020-06-27 07:54:23
exe             = ./mdtest -n 60000 -t -w 3901 -e 3901 -P -N 1 -F -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall -T -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-hard-stat.txt
time            = 28.243523
score           = 106.697030
t_delta         = 28.2638
t_end           = 2020-06-27 07:54:51

[mdtest-easy-delete]
t_start         = 2020-06-27 07:54:51
exe             = ./mdtest -n 2300000 -u -L -F -P -N 1 -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-easy-stonewall -r -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-easy-delete.txt
time            = 831.018521
score           = 32.592655
t_delta         = 835.6373
t_end           = 2020-06-27 08:08:47

[mdtest-hard-read]
t_start         = 2020-06-27 08:08:47
exe             = ./mdtest -n 60000 -t -w 3901 -e 3901 -P -N 1 -F -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall -E -X -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-hard-read.txt
time            = 58.171566
score           = 51.803660
t_delta         = 58.1963
t_end           = 2020-06-27 08:09:45

[mdtest-hard-delete]
t_start         = 2020-06-27 08:09:45
exe             = ./mdtest -n 60000 -t -w 3901 -e 3901 -P -N 1 -F -d /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard -x /gpfs/hal/io500/datafiles/2020.06.27-06.53.38-app/mdtest-hard-stonewall -r -a POSIX
result-file     = ./results/2020.06.27-06.53.38-app/mdtest-hard-delete.txt
time            = 6378.708898
score           = 0.472431
t_delta         = 6378.8230
t_end           = 2020-06-27 09:56:04

[SCORE]
MD              = 34.153035
BW              = 3.625219
SCORE           = 11.127094 
hash            = 5D1B96AD
; END 2020-06-27 09:56:05
result_summary
IO500 version io500-isc20-5-gc155836d31c1
[RESULT]       ior-easy-write       14.240362 GiB/s  : time 391.554 seconds
[RESULT]    mdtest-easy-write       74.672307 kIOPS : time 365.516 seconds
[RESULT]       ior-hard-write        0.479759 GiB/s  : time 353.527 seconds
[RESULT]    mdtest-hard-write        1.739822 kIOPS : time 1732.245 seconds
[RESULT]                 find     1553.046946 kIOPS : time 19.394 seconds
[RESULT]        ior-easy-read       12.737400 GiB/s  : time 437.545 seconds
[RESULT]     mdtest-easy-stat      107.798979 kIOPS : time 251.275 seconds
[RESULT]        ior-hard-read        1.984777 GiB/s  : time 85.456 seconds
[RESULT]     mdtest-hard-stat      106.697030 kIOPS : time 28.264 seconds
[RESULT]   mdtest-easy-delete       32.592655 kIOPS : time 835.637 seconds
[RESULT]     mdtest-hard-read       51.803660 kIOPS : time 58.196 seconds
[RESULT]   mdtest-hard-delete        0.472431 kIOPS : time 6378.823 seconds
[SCORE] Bandwidth 3.625219 GB/s : IOPS 34.153035 kiops : TOTAL 11.127094