VastTest

Institution Janelia Research Campus, HHMI
Client Procs Per Node
Client Operating System Scientific-Linux
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.9.1.el7.x86_64

DATA SERVER

Storage Type Flash
Volatile Memory 64GB
Storage Interface NVMe
Network 100GbE
Software Version 2.0.7
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 620
Metadata Nodes 4
Metadata Storage Devices 48
Data Nodes 1
Data Storage Devices 132

METADATA

Easy Write 12.61 kIOP/s
Easy Stat 88.41 kIOP/s
Easy Delete 10.02 kIOP/s
Hard Write 2.25 kIOP/s
Hard Read 11.94 kIOP/s
Hard Stat 21.89 kIOP/s
Hard Delete 1.86 kIOP/s

Submitted Files

ior_easy_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Wed Dec 18 13:00:31 2019
Command line        : /io500/io500-sc19/bin/ior -r -R -t 1024k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy/ior_file_easy -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy/stonewall
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Wed Dec 18 13:00:31 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy
FS                  : 15641.3 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 14077.15 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 15477988392960000.
WARNING: Stat() of aggregate file size      = 3055030370304.
WARNING: Using actual aggregate bytes moved = 3055030370304.
read      15253      10158080000 1024.00    0.147648   190.86     0.007157   191.01     0   
Max Read:  15252.78 MiB/sec (15993.70 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        15252.78   15252.78   15252.78       0.00   15252.78   15252.78   15252.78       0.00  191.01459         NA            NA     0   1488  62    1   1     1        1         0    0      1 10401873920000  1048576 2913504.0 POSIX      0
Finished            : Wed Dec 18 13:03:42 2019
ior_easy_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Wed Dec 18 11:50:53 2019
Command line        : /io500/io500-sc19/bin/ior -w -t 1024k -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy/ior_file_easy -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Wed Dec 18 11:50:53 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy
FS                  : 24726.0 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 14077.15 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
stonewalling pairs accessed min: 741 max: 1958 -- min data: 0.7 GiB mean data: 1.2 GiB time: 301.0s
WARNING: Expected aggregate file size       = 15477988392960000.
WARNING: Stat() of aggregate file size      = 3055030370304.
WARNING: Using actual aggregate bytes moved = 3055030370304.
WARNING: maybe caused by deadlineForStonewalling
write     2815.71    10158080000 1024.00    0.669662   1033.98    0.080879   1034.73    0   
Max Write: 2815.71 MiB/sec (2952.48 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2815.71    2815.71    2815.71       0.00    2815.71    2815.71    2815.71       0.00 1034.73209     301.02       6059.06     0   1488  62    1   1     1        1         0    0      1 10401873920000  1048576 2913504.0 POSIX      0
Finished            : Wed Dec 18 12:08:08 2019
ior_hard_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Wed Dec 18 13:04:57 2019
Command line        : /io500/io500-sc19/bin/ior -r -R -s 1900000 -a POSIX --posix.odirect -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard/IOR_file -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard/stonewall
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Wed Dec 18 13:04:57 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard
FS                  : 15641.3 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 120.87 TiB

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
WARNING: Expected aggregate file size       = 132901017600000.
WARNING: Stat() of aggregate file size      = 620158116864.
WARNING: Using actual aggregate bytes moved = 620158116864.
read      3891       45.91      45.91      0.029437   151.95     0.000707   151.98     0   
Max Read:  3891.43 MiB/sec (4080.46 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         3891.43    3891.43    3891.43       0.00   86803.42   86803.42   86803.42       0.00  151.98258         NA            NA     0   1488  62    1   0     1        1         0    0 1900000    47008    47008  591428.9 POSIX      0
Finished            : Wed Dec 18 13:07:29 2019
ior_hard_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Wed Dec 18 12:14:55 2019
Command line        : /io500/io500-sc19/bin/ior -w -s 1900000 -a POSIX --posix.odirect -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard/IOR_file -O stoneWallingStatusFile=/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux h09u01.int.janelia.org
TestID              : 0
StartTime           : Wed Dec 18 12:14:55 2019
Path                : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard
FS                  : 15637.6 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.7%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 1900000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
tasks               : 1488
clients per node    : 62
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 120.87 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s) iter
------    ---------  ---------- ---------  --------   --------   --------   -------- ----
stonewalling pairs accessed min: 3257 max: 8866 -- min data: 0.1 GiB mean data: 0.3 GiB time: 300.2s
WARNING: Expected aggregate file size       = 132901017600000.
WARNING: Stat() of aggregate file size      = 620158116864.
WARNING: Using actual aggregate bytes moved = 620158116864.
WARNING: maybe caused by deadlineForStonewalling
write     1192.79    45.91      45.91      0.041408   495.79     0.000863   495.84     0   
Max Write: 1192.79 MiB/sec (1250.73 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        1192.79    1192.79    1192.79       0.00   26606.75   26606.75   26606.75       0.00  495.83681     300.19       1298.08     0   1488  62    1   0     1        1         0    0 1900000    47008    47008  591428.9 POSIX      0
Finished            : Wed Dec 18 12:23:11 2019
mdtest_easy_delete
-- started at 12/18/2019 13:08:21 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-r' '-F' '-P' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_easy' '-n' '900000' '-u' '-L' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_easy-stonewall' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 15641.3 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1339200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      10015.937      10015.924      10015.931          0.002
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.815          0.815          0.815          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        484.465        484.465        484.465          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.226          1.226          1.226          0.000
-- finished at 12/18/2019 13:16:28 --

mdtest_easy_stat
-- started at 12/18/2019 13:03:52 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-T' '-F' '-P' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_easy' '-n' '900000' '-u' '-L' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_easy-stonewall' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 15641.3 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1339200000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      88414.110      88413.542      88413.831          0.098
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         54.883         54.882         54.882          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/18/2019 13:04:47 --

mdtest_easy_write
-- started at 12/18/2019 12:08:18 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-C' '-F' '-P' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_easy' '-n' '900000' '-u' '-L' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_easy-stonewall' '-N' '1' '-W' '300'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 15637.4 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1339200000 files
Continue stonewall hit min: 1827 max: 3261 avg: 2805.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      12606.164      12606.135      12606.155          0.004
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA      13937.095             NA
   Tree creation             :          1.458          1.458          1.458          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        384.921        384.920        384.921          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.495             NA
   Tree creation             :          0.686          0.686          0.686          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/18/2019 12:14:44 --

mdtest_hard_delete
-- started at 12/18/2019 13:17:50 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-r' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 13172.5 TiB   Used FS: 0.1%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       1858.130       1858.129       1858.130          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.387          2.387          2.387          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        397.200        397.199        397.199          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.419          0.419          0.419          0.000
-- finished at 12/18/2019 13:24:28 --

mdtest_hard_read
-- started at 12/18/2019 13:16:38 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-X' '-E' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 13168.0 TiB   Used FS: 0.1%   Inodes: 1377.5 Mi   Used Inodes: 0.4%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      11941.368      11941.343      11941.355          0.006
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :         61.806         61.806         61.806          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/18/2019 13:17:40 --

mdtest_hard_stat
-- started at 12/18/2019 13:07:38 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-T' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 15641.3 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.8%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      21890.585      21890.500      21890.549          0.017
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         33.715         33.715         33.715          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/18/2019 13:08:12 --

mdtest_hard_write
-- started at 12/18/2019 12:23:21 --

mdtest-3.3.0+dev was launched with 1488 total task(s) on 24 node(s)
Command line used: /io500/io500-sc19/bin/mdtest '-C' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard' '-n' '950000' '-x' '/io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1' '-W' '300'
Path: /io500/io500-sc19/datafiles/io500.2019.12.18-11.50.43
FS: 15638.9 TiB   Used FS: 0.0%   Inodes: 1377.5 Mi   Used Inodes: 0.7%

Nodemap: 111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2128 Shifting ranks by 62 for each phase.
1488 tasks, 1413600000 files
Continue stonewall hit min: 454 max: 496 avg: 472.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       2250.101       2250.099       2250.100          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       2338.196             NA
   Tree creation             :        252.016        252.016        252.016          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        328.007        328.007        328.007          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.628             NA
   Tree creation             :          0.004          0.004          0.004          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 12/18/2019 12:28:50 --

result_summary
[RESULT] BW   phase 1            ior_easy_write                2.750 GB/s : time 1034.73 seconds
[RESULT] IOPS phase 1         mdtest_easy_write               12.606 kiops : time 384.92 seconds
[RESULT] BW   phase 2            ior_hard_write                1.165 GB/s : time 495.84 seconds
[RESULT] IOPS phase 2         mdtest_hard_write                2.250 kiops : time 328.01 seconds
[RESULT] IOPS phase 3                      find                2.950 kiops : time 1891.04 seconds
[RESULT] BW   phase 3             ior_easy_read               14.895 GB/s : time 191.01 seconds
[RESULT] IOPS phase 4          mdtest_easy_stat               88.414 kiops : time  54.88 seconds
[RESULT] BW   phase 4             ior_hard_read                3.800 GB/s : time 151.98 seconds
[RESULT] IOPS phase 5          mdtest_hard_stat               21.891 kiops : time  33.72 seconds
[RESULT] IOPS phase 6        mdtest_easy_delete               10.016 kiops : time 484.46 seconds
[RESULT] IOPS phase 7          mdtest_hard_read               11.941 kiops : time  61.81 seconds
[RESULT] IOPS phase 8        mdtest_hard_delete                1.858 kiops : time 408.59 seconds
[SCORE] Bandwidth 3.66937 GB/s : IOPS 8.80088 kiops : TOTAL 5.68275