VastTest

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 64GB
Storage Interface NVMe
Network 100GbE
Software Version 2.0.7
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 620
Metadata Nodes 4
Metadata Storage Devices 48
Data Nodes 1
Data Storage Devices 132

METADATA

Easy Write 12.80 kIOP/s
Easy Stat 82.42 kIOP/s
Easy Delete 10.24 kIOP/s
Hard Write 2.27 kIOP/s
Hard Read 11.88 kIOP/s
Hard Stat 22.25 kIOP/s
Hard Delete 2.02 kIOP/s

Submitted Files

ior_easy_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Dec 19 06:45:36 2019
Command line        : /io500/io500-sc19/bin/ior -r -R -t 1024k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy/ior_file_easy -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy/stonewall
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Thu Dec 19 06:45:36 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy
FS                  : 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 14077.15 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 15477988392960000.
WARNING: Stat() of aggregate file size      = 3574603972608.
WARNING: Using actual aggregate bytes moved = 3574603972608.
read      12933      10158080000 1024.00    0.198109   263.39     0.007801   263.60     0   
Max Read:  12932.65 MiB/sec (13560.86 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        12932.65   12932.65   12932.65       0.00   12932.65   12932.65   12932.65       0.00  263.59710         NA            NA     0   1488  62    1   1     1        1         0    0      1 10401873920000  1048576 3409008.0 POSIX      0
Finished            : Thu Dec 19 06:50:00 2019
ior_easy_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Dec 19 06:01:26 2019
Command line        : /io500/io500-sc19/bin/ior -w -t 1024k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy/ior_file_easy -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Thu Dec 19 06:01:26 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy
FS                  : 24861.7 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 14077.15 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
stonewalling pairs accessed min: 794 max: 2291 -- min data: 0.8 GiB mean data: 1.4 GiB time: 301.6s
WARNING: Expected aggregate file size       = 15477988392960000.
WARNING: Stat() of aggregate file size      = 3574603972608.
WARNING: Using actual aggregate bytes moved = 3574603972608.
WARNING: maybe caused by deadlineForStonewalling
write     2959.36    10158080000 1024.00    0.660991   1151.17    0.116585   1151.94    0   
Max Write: 2959.36 MiB/sec (3103.11 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2959.36    2959.36    2959.36       0.00    2959.36    2959.36    2959.36       0.00 1151.94254     301.61       6840.71     0   1488  62    1   1     1        1         0    0      1 10401873920000  1048576 3409008.0 POSIX      0
Finished            : Thu Dec 19 06:20:38 2019
ior_hard_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Dec 19 06:51:20 2019
Command line        : /io500/io500-sc19/bin/ior -r -R -s 1900000 -a POSIX --posix.odirect -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard/IOR_file -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard/stonewall
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Thu Dec 19 06:51:20 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard
FS                  : 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 120.87 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 132901017600000.
WARNING: Stat() of aggregate file size      = 609595983360.
WARNING: Using actual aggregate bytes moved = 609595983360.
read      3845       45.91      45.91      0.037358   151.18     0.000692   151.21     0   
Max Read:  3844.59 MiB/sec (4031.35 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         3844.59    3844.59    3844.59       0.00   85758.77   85758.77   85758.77       0.00  151.21392         NA            NA     0   1488  62    1   0     1        1         0    0 1900000    47008    47008  581356.1 POSIX      0
Finished            : Thu Dec 19 06:53:51 2019
ior_hard_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Dec 19 06:27:28 2019
Command line        : /io500/io500-sc19/bin/ior -w -s 1900000 -a POSIX --posix.odirect -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard/IOR_file -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Thu Dec 19 06:27:28 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard
FS                  : 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.7%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 120.87 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
stonewalling pairs accessed min: 3133 max: 8715 -- min data: 0.1 GiB mean data: 0.3 GiB time: 300.1s
WARNING: Expected aggregate file size       = 132901017600000.
WARNING: Stat() of aggregate file size      = 609595983360.
WARNING: Using actual aggregate bytes moved = 609595983360.
WARNING: maybe caused by deadlineForStonewalling
write     1128.72    45.91      45.91      0.034646   515.02     0.000768   515.06     0   
Max Write: 1128.72 MiB/sec (1183.55 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        1128.72    1128.72    1128.72       0.00   25177.62   25177.62   25177.62       0.00  515.05744     300.12       1269.67     0   1488  62    1   0     1        1         0    0 1900000    47008    47008  581356.1 POSIX      0
Finished            : Thu Dec 19 06:36:04 2019
mdtest_easy_delete
-- started at 12/19/2019 06:54:45 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-r' '-F' '-P' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_easy' '-n' '900000' '-u' '-L' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_easy-stonewall' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1339200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       9628.274       9628.259       9628.265          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.116          0.116          0.116          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        503.508        503.507        503.508          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          8.608          8.608          8.608          0.000
-- finished at 12/19/2019 07:03:18 --

mdtest_easy_stat
-- started at 12/19/2019 06:50:11 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-T' '-F' '-P' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_easy' '-n' '900000' '-u' '-L' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_easy-stonewall' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1339200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      84276.382      84275.774      84276.118          0.087
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         57.524         57.524         57.524          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/19/2019 06:51:09 --

mdtest_easy_write
-- started at 12/19/2019 06:20:48 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-C' '-F' '-P' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_easy' '-n' '900000' '-u' '-L' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_easy-stonewall' '-N' '1' '-W' '300'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1339200000 files
Continue stonewall hit min: 1805 max: 3258 avg: 2783.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      12464.924      12464.907      12464.917          0.003
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      13829.146             NA
   Tree creation             :          1.480          1.480          1.480          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        388.924        388.924        388.924          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.495             NA
   Tree creation             :          0.676          0.676          0.676          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/19/2019 06:27:17 --

mdtest_hard_delete
-- started at 12/19/2019 07:04:42 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-r' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 23650.0 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       1792.203       1792.199       1792.200          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         10.113         10.113         10.113          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        405.999        405.999        405.999          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.099          0.099          0.099          0.000
-- finished at 12/19/2019 07:11:28 --

mdtest_hard_read
-- started at 12/19/2019 07:03:28 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-X' '-E' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 22195.6 TiB   Used FS: 0.1%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      11518.788      11518.756      11518.772          0.006
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         63.169         63.169         63.169          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/19/2019 07:04:32 --

mdtest_hard_stat
-- started at 12/19/2019 06:54:01 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-T' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      20814.124      20814.018      20814.067          0.021
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         34.959         34.959         34.959          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/19/2019 06:54:36 --

mdtest_hard_write
-- started at 12/19/2019 06:36:14 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-C' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1' '-W' '300'
Path: /io500/io500-sc19/datafiles/io500.2019.12.19-06.01.15
FS: 24861.8 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.7%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files
Continue stonewall hit min: 452 max: 489 avg: 469.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2245.862       2245.859       2245.860          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2319.415             NA
   Tree creation             :        256.344        256.344        256.344          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        323.988        323.988        323.988          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        301.004             NA
   Tree creation             :          0.004          0.004          0.004          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/19/2019 06:41:38 --

result_summary
[RESULT] BW   phase 1            ior_easy_write                2.890 GB/s : time 1151.94 seconds
[RESULT] IOPS phase 1         mdtest_easy_write               12.465 kiops : time 388.92 seconds
[RESULT] BW   phase 2            ior_hard_write                1.102 GB/s : time 515.06 seconds
[RESULT] IOPS phase 2         mdtest_hard_write                2.246 kiops : time 323.99 seconds
[RESULT] IOPS phase 3                      find               21.220 kiops : time 228.99 seconds
[RESULT] BW   phase 3             ior_easy_read               12.630 GB/s : time 263.60 seconds
[RESULT] IOPS phase 4          mdtest_easy_stat               84.276 kiops : time  57.52 seconds
[RESULT] BW   phase 4             ior_hard_read                3.755 GB/s : time 151.21 seconds
[RESULT] IOPS phase 5          mdtest_hard_stat               20.814 kiops : time  34.96 seconds
[RESULT] IOPS phase 6        mdtest_easy_delete                9.628 kiops : time 503.51 seconds
[RESULT] IOPS phase 7          mdtest_hard_read               11.519 kiops : time  63.17 seconds
[RESULT] IOPS phase 8        mdtest_hard_delete                1.792 kiops : time 415.92 seconds
[SCORE] Bandwidth 3.50587 GB/s : IOPS 10.9529 kiops : TOTAL 6.19673