AI400X

Institution DDN
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 8.1
Client Kernel Version 4.18.0-147.8.1.el8_1.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 3.84TB
Storage Interface NVMe
Network IB-EDR
Software Version 2.13.54
OS Version CentOS7.7

INFORMATION

Client Nodes 40
Client Total Procs 1,280
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 4
Data Storage Devices 10

METADATA

Easy Write 355.80 kIOP/s
Easy Stat 1,114.09 kIOP/s
Easy Delete 242.87 kIOP/s
Hard Write 130.61 kIOP/s
Hard Read 356.29 kIOP/s
Hard Stat 1,100.21 kIOP/s
Hard Delete 91.94 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 16:18:14 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/stonewall -t 16m -b 70000m -F -r -R -a POSIX --posix.odirect
Machine             : Linux ec01
TestID              : 0
StartTime           : Tue Jul 14 16:18:14 2020
Path                : /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy
FS                  : 94.9 TiB   Used FS: 65.6%   Inodes: 926.9 Mi   Used Inodes: 17.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1280
clients per node    : 32
repetitions         : 1
xfersize            : 16 MiB
blocksize           : 68.36 GiB
aggregate filesize  : 85.45 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 93952409600000.
WARNING: Stat() of aggregate file size      = 30623116820480.
WARNING: Using actual aggregate bytes moved = 30623116820480.
read      93439      5841       0.071430    71680000   16384      0.032463   312.51     0.011060   312.55     0   
Max Read:  93439.27 MiB/sec (97978.18 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        93439.27   93439.27   93439.27       0.00    5839.95    5839.95    5839.95       0.00  312.55037         NA            NA     0   1280  32    1   1     1        1         0    0      1 73400320000 16777216 29204480.0 POSIX      0
Finished            : Tue Jul 14 16:23:27 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 15:48:15 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/stonewall -t 16m -b 70000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
Machine             : Linux ec01
TestID              : 0
StartTime           : Tue Jul 14 15:48:15 2020
Path                : /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy
FS                  : 94.9 TiB   Used FS: 33.1%   Inodes: 768.0 Mi   Used Inodes: 0.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1280
clients per node    : 32
repetitions         : 1
xfersize            : 16 MiB
blocksize           : 68.36 GiB
aggregate filesize  : 85.45 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 836 max: 1426 -- min data: 13.1 GiB mean data: 17.5 GiB time: 300.2s
WARNING: Expected aggregate file size       = 93952409600000.
WARNING: Stat() of aggregate file size      = 30623116820480.
WARNING: Using actual aggregate bytes moved = 30623116820480.
WARNING: maybe caused by deadlineForStonewalling
write     74752      4674       0.077178    71680000   16384      0.102473   390.56     0.025541   390.69     0   
Max Write: 74751.89 MiB/sec (78383.04 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       74751.89   74751.89   74751.89       0.00    4671.99    4671.99    4671.99       0.00  390.68550     300.20      76453.90     0   1280  32    1   1     1        1         0    0      1 73400320000 16777216 29204480.0 POSIX      0
Finished            : Tue Jul 14 15:54:46 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 16:25:16 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/file -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 55000 -r -R -a POSIX
Machine             : Linux ec01
TestID              : 0
StartTime           : Tue Jul 14 16:25:16 2020
Path                : /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard
FS                  : 94.9 TiB   Used FS: 65.6%   Inodes: 926.9 Mi   Used Inodes: 17.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 55000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1280
clients per node    : 32
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 3.01 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 55000 max: 55000 -- min data: 2.4 GiB mean data: 2.4 GiB time: 38.4s
read      82070      1832097    38.43       45.91      45.91      0.021920   38.43      0.008016   38.46      0   
Max Read:  82069.93 MiB/sec (86056.56 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        82069.93   82069.93   82069.93       0.00 1830678.97 1830678.97 1830678.97       0.00   38.45568      38.43      82135.17     0   1280  32    1   0     1        1         0    0  55000    47008    47008 3156054.8 POSIX      0
Finished            : Tue Jul 14 16:25:55 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Tue Jul 14 16:00:28 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/file -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 55000 -w -D 300 -a POSIX
Machine             : Linux ec01
TestID              : 0
StartTime           : Tue Jul 14 16:00:28 2020
Path                : /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard
FS                  : 94.9 TiB   Used FS: 62.4%   Inodes: 884.0 Mi   Used Inodes: 13.1%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 55000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 40
tasks               : 1280
clients per node    : 32
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 3.01 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 36377 max: 55000 -- min data: 1.6 GiB mean data: 2.4 GiB time: 300.2s
write     5272       117599     300.45      45.91      45.91      0.010478   598.65     0.012077   598.67     0   
Max Write: 5271.79 MiB/sec (5527.87 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        5271.79    5271.79    5271.79       0.00  117594.25  117594.25  117594.25       0.00  598.66874     300.18      10460.16     0   1280  32    1   0     1        1         0    0  55000    47008    47008 3156054.8 POSIX      0
Finished            : Tue Jul 14 16:10:30 2020
mdtest-easy-delete
-- started at 07/14/2020 16:26:36 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '95000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 65.6%   Inodes: 926.9 Mi   Used Inodes: 17.1%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 121600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     242869.889     242869.808     242869.871          0.012
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.632          0.632          0.632          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        500.680        500.680        500.680          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.583          1.583          1.583          0.000
-- finished at 07/14/2020 16:35:03 --

mdtest-easy-stat
-- started at 07/14/2020 16:23:27 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '95000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 65.6%   Inodes: 926.9 Mi   Used Inodes: 17.1%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 121600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    1114090.100    1114089.208    1114089.749          0.174
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        109.147        109.147        109.147          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 16:25:16 --

mdtest-easy-write
-- started at 07/14/2020 15:54:46 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '95000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 62.4%   Inodes: 768.0 Mi   Used Inodes: 0.0%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 121600000 files
Continue stonewall hit min: 77261 max: 95000 avg: 92974.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     355804.221     355804.023     355804.089          0.040
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     397789.785             NA
   Tree creation             :          1.182          1.182          1.182          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        341.761        341.761        341.761          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.171             NA
   Tree creation             :          0.846          0.846          0.846          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 16:00:28 --

mdtest-hard-delete
-- started at 07/14/2020 16:37:09 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '200000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 65.6%   Inodes: 810.9 Mi   Used Inodes: 5.3%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 256000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      91940.229      91940.203      91940.219          0.005
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.217          0.217          0.217          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        489.389        489.389        489.389          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          4.608          4.608          4.608          0.000
-- finished at 07/14/2020 16:45:24 --

mdtest-hard-read
-- started at 07/14/2020 16:35:03 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '200000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 65.6%   Inodes: 810.9 Mi   Used Inodes: 5.3%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 256000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     356286.792     356286.242     356286.411          0.085
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        126.288        126.287        126.288          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 16:37:09 --

mdtest-hard-stat
-- started at 07/14/2020 16:25:55 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '200000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 65.6%   Inodes: 926.9 Mi   Used Inodes: 17.1%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 256000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    1100207.230    1100206.187    1100206.763          0.214
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         40.896         40.896         40.896          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 16:26:36 --

mdtest-hard-write
-- started at 07/14/2020 16:10:30 --

mdtest-3.3.0+dev was launched with 1280 total task(s) on 40 node(s)
Command line used: ./mdtest '-n' '200000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard' '-x' '/ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /ai400/datafiles/2020.07.14-14.55.35-app
FS: 94.9 TiB   Used FS: 65.6%   Inodes: 884.0 Mi   Used Inodes: 13.1%

Nodemap: 11111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 32 for each phase.
1280 tasks, 256000000 files
Continue stonewall hit min: 29935 max: 35152 avg: 32764.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     130609.573     130609.128     130609.536          0.020
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     139831.099             NA
   Tree creation             :          8.333          8.333          8.333          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        344.498        344.497        344.497          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.926             NA
   Tree creation             :          0.120          0.120          0.120          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/14/2020 16:16:14 --

result
version         = io500-isc20_v4-2-g20efd247106c
config-hash     = 3DED1729
result-dir      = ./results/2020.07.14-14.55.35-app
; START 2020-07-14 15:48:15


[ior-easy-write]
t_start         = 2020-07-14 15:48:15
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/stonewall -t 16m -b 70000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
throughput-stonewall = 73434.46
score           = 72.999893
t_delta         = 390.7892
t_end           = 2020-07-14 15:54:46

[mdtest-easy-write]
t_start         = 2020-07-14 15:54:46
exe             = ./mdtest -n 95000 -u -L -F -P -N 1 -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 397.789785
score           = 355.804093
t_delta         = 342.6499
t_end           = 2020-07-14 16:00:28

[timestamp]
t_start         = 2020-07-14 16:00:28
t_delta         = 0.0162
t_end           = 2020-07-14 16:00:28

[ior-hard-write]
t_start         = 2020-07-14 16:00:28
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/file -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 55000 -w -D 300 -a POSIX
throughput-stonewall = 6556.06
score           = 5.148231
t_delta         = 601.2087
t_end           = 2020-07-14 16:10:30

[mdtest-hard-write]
t_start         = 2020-07-14 16:10:30
exe             = ./mdtest -n 200000 -t -w 3901 -e 3901 -P -N 1 -F -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 139.831099
score           = 130.609522
t_delta         = 344.6678
t_end           = 2020-07-14 16:16:14

[find]
t_start         = 2020-07-14 16:16:14
exe             = ./pfind /ai400/datafiles/2020.07.14-14.55.35-app -newer /ai400/datafiles/2020.07.14-14.55.35-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 2028828
total-files     = 166595846
score           = 1386.838276
t_delta         = 120.1294
t_end           = 2020-07-14 16:18:14

[ior-easy-read]
t_start         = 2020-07-14 16:18:14
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-easy/stonewall -t 16m -b 70000m -F -r -R -a POSIX --posix.odirect
score           = 91.249291
t_delta         = 312.6579
t_end           = 2020-07-14 16:23:27

[mdtest-easy-stat]
t_start         = 2020-07-14 16:23:27
exe             = ./mdtest -n 95000 -u -L -F -P -N 1 -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy-stonewall -T -a POSIX
score           = 1114.090096
t_delta         = 109.1534
t_end           = 2020-07-14 16:25:16

[ior-hard-read]
t_start         = 2020-07-14 16:25:16
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/file -O stoneWallingStatusFile=/ai400/datafiles/2020.07.14-14.55.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 55000 -r -R -a POSIX
score           = 80.146414
t_delta         = 38.6421
t_end           = 2020-07-14 16:25:55

[mdtest-hard-stat]
t_start         = 2020-07-14 16:25:55
exe             = ./mdtest -n 200000 -t -w 3901 -e 3901 -P -N 1 -F -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall -T -a POSIX
score           = 1100.207072
t_delta         = 40.9010
t_end           = 2020-07-14 16:26:36

[mdtest-easy-delete]
t_start         = 2020-07-14 16:26:36
exe             = ./mdtest -n 95000 -u -L -F -P -N 1 -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-easy-stonewall -r -a POSIX
score           = 242.869871
t_delta         = 507.4339
t_end           = 2020-07-14 16:35:03

[mdtest-hard-read]
t_start         = 2020-07-14 16:35:03
exe             = ./mdtest -n 200000 -t -w 3901 -e 3901 -P -N 1 -F -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 356.286321
t_delta         = 126.3284
t_end           = 2020-07-14 16:37:09

[mdtest-hard-delete]
t_start         = 2020-07-14 16:37:09
exe             = ./mdtest -n 200000 -t -w 3901 -e 3901 -P -N 1 -F -d /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard -x /ai400/datafiles/2020.07.14-14.55.35-app/mdtest-hard-stonewall -r -a POSIX
score           = 91.940216
t_delta         = 494.0511
t_end           = 2020-07-14 16:45:24

[SCORE]
MD              = 397.910667
BW              = 40.716782
SCORE           = 127.285671 
hash            = 7FA816F9
; END 2020-07-14 16:45:24
result_summary
IO500 version io500-isc20_v4-2-g20efd247106c
[RESULT]       ior-easy-write       72.999893 GiB/s  : time 390.789 seconds
[RESULT]    mdtest-easy-write      355.804093 kIOPS : time 342.650 seconds
[RESULT]       ior-hard-write        5.148231 GiB/s  : time 601.209 seconds
[RESULT]    mdtest-hard-write      130.609522 kIOPS : time 344.668 seconds
[RESULT]                 find     1386.838276 kIOPS : time 120.129 seconds
[RESULT]        ior-easy-read       91.249291 GiB/s  : time 312.658 seconds
[RESULT]     mdtest-easy-stat     1114.090096 kIOPS : time 109.153 seconds
[RESULT]        ior-hard-read       80.146414 GiB/s  : time 38.642 seconds
[RESULT]     mdtest-hard-stat     1100.207072 kIOPS : time 40.901 seconds
[RESULT]   mdtest-easy-delete      242.869871 kIOPS : time 507.434 seconds
[RESULT]     mdtest-hard-read      356.286321 kIOPS : time 126.328 seconds
[RESULT]   mdtest-hard-delete       91.940216 kIOPS : time 494.051 seconds
[SCORE] Bandwidth 40.716782 GB/s : IOPS 397.910667 kiops : TOTAL 127.285671