MVS10POP2

Institution Joint Supercomputer Center of the Russian Academy of Sciences
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7.1908
Client Kernel Version 3.10.0-1062.1.1.el7.x86_64

DATA SERVER

Storage Type SSD-NVMe
Volatile Memory 187G
Storage Interface NVMe
Network Omnipath
Software Version 2.13
OS Version 7.7.1908

INFORMATION

Client Nodes 54
Client Total Procs 864
Metadata Nodes 50
Metadata Storage Devices 1
Data Nodes 50
Data Storage Devices 1

METADATA

Easy Write 319.52 kIOP/s
Easy Stat 2,171.38 kIOP/s
Easy Delete 526.14 kIOP/s
Hard Write 62.72 kIOP/s
Hard Read 250.47 kIOP/s
Hard Stat 348.52 kIOP/s
Hard Delete 59.04 kIOP/s

Submitted Files

ior-easy-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Oct 30 02:13:17 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-easy.stonewall -t 2m -b 9920000m -F -r -R -a POSIX --posix.odirect
Machine             : Linux s01p001
TestID              : 0
StartTime           : Fri Oct 30 02:13:17 2020
Path                : /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy.00000000
FS                  : 180.1 TiB   Used FS: 27.2%   Inodes: 1224.7 Mi   Used Inodes: 47.3%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 54
tasks               : 756
clients per node    : 14
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 7152.10 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 7863816683520000
WARNING: Stat() of aggregate file size      = 44674723086336
WARNING: Using actual aggregate bytes moved = 44674723086336
read      125678     62841      0.000068    10158080000 2048.00    0.013666   338.99     0.013348   339.00     0   
ior-easy-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Oct 30 01:00:48 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-easy.stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
Machine             : Linux s01p001
TestID              : 0
StartTime           : Fri Oct 30 01:00:48 2020
Path                : /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy.00000000
FS                  : 180.1 TiB   Used FS: 0.0%   Inodes: 727.6 Mi   Used Inodes: 0.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 54
tasks               : 756
clients per node    : 14
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 7152.10 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 19561 max: 28178 -- min data: 38.2 GiB mean data: 42.3 GiB time: 300.0s
WARNING: Expected aggregate file size       = 7863816683520000
WARNING: Stat() of aggregate file size      = 44674723086336
WARNING: Using actual aggregate bytes moved = 44674723086336
WARNING: Maybe caused by deadlineForStonewalling
write     100662     50333      0.000077    10158080000 2048.00    0.008897   423.23     0.007259   423.25     0   
ior-hard-read
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Oct 30 02:22:58 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 250000 -r -R -a MPIIO
Machine             : Linux s01p001
TestID              : 0
StartTime           : Fri Oct 30 02:22:58 2020
Path                : /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file
FS                  : 180.1 TiB   Used FS: 27.2%   Inodes: 1224.7 Mi   Used Inodes: 47.3%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 250000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 54
tasks               : 756
clients per node    : 14
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 8.08 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 250000 max: 250000 -- min data: 10.9 GiB mean data: 10.9 GiB time: 472.1s
read      17940      400289     472.15      45.91      45.91      0.139807   472.16     0.016423   472.29     0   
ior-hard-write
IOR-3.4.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Oct 30 01:36:17 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 250000 -w -D 300 -a MPIIO
Machine             : Linux s01p001
TestID              : 0
StartTime           : Fri Oct 30 01:36:17 2020
Path                : /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file
FS                  : 180.1 TiB   Used FS: 22.6%   Inodes: 1224.7 Mi   Used Inodes: 40.6%

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 250000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 54
tasks               : 756
clients per node    : 14
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 8.08 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 166904 max: 250000 -- min data: 7.3 GiB mean data: 9.4 GiB time: 300.0s
write     20420      455563     364.34      45.91      45.91      0.070184   414.87     0.009173   414.94     0   
mdtest-easy-delete
-- started at 10/30/2020 02:35:04 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-easy' '-x' './results/2020.10.30-00.59.02/mdtest-easy.stonewall' '-r' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 180.1 TiB   Used FS: 27.2%   Inodes: 1224.7 Mi   Used Inodes: 47.3%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     526144.738     526140.301     526142.414          0.996
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.547          0.547          0.547          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        990.541        990.533        990.537          0.002
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.827          1.827          1.827          0.000
-- finished at 10/30/2020 02:51:39 --

mdtest-easy-stat
-- started at 10/30/2020 02:18:57 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-easy' '-x' './results/2020.10.30-00.59.02/mdtest-easy.stonewall' '-T' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 180.1 TiB   Used FS: 27.2%   Inodes: 1224.7 Mi   Used Inodes: 47.3%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    2171375.261    2171356.910    2171365.719          4.101
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        240.018        240.016        240.017          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/30/2020 02:22:58 --

mdtest-easy-write
-- started at 10/30/2020 01:07:52 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-easy' '-x' './results/2020.10.30-00.59.02/mdtest-easy.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 180.1 TiB   Used FS: 22.6%   Inodes: 727.6 Mi   Used Inodes: 0.0%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files
Continue stonewall hit min: 359301 max: 689370 avg: 546439.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     319521.106     319518.505     319519.760          0.589
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    1375456.403             NA
   Tree creation             :          1.784          1.784          1.784          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       1631.091       1631.078       1631.084          0.003
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.343             NA
   Tree creation             :          0.561          0.561          0.561          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/30/2020 01:36:17 --

mdtest-hard-delete
-- started at 10/30/2020 02:57:25 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-hard' '-x' './results/2020.10.30-00.59.02/mdtest-hard.stonewall' '-r' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 180.1 TiB   Used FS: 27.2%   Inodes: 727.6 Mi   Used Inodes: 11.3%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      59036.743      59036.256      59036.493          0.110
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.325          0.325          0.325          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       1462.474       1462.462       1462.468          0.003
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          3.074          3.074          3.074          0.000
-- finished at 10/30/2020 03:21:54 --

mdtest-hard-read
-- started at 10/30/2020 02:51:39 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-hard' '-x' './results/2020.10.30-00.59.02/mdtest-hard.stonewall' '-E' '-X' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 180.1 TiB   Used FS: 27.2%   Inodes: 727.6 Mi   Used Inodes: 11.3%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     250473.288     250471.179     250472.195          0.473
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        344.706        344.703        344.705          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/30/2020 02:57:25 --

mdtest-hard-stat
-- started at 10/30/2020 02:30:56 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-hard' '-x' './results/2020.10.30-00.59.02/mdtest-hard.stonewall' '-T' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 180.1 TiB   Used FS: 27.2%   Inodes: 1224.7 Mi   Used Inodes: 47.3%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     348524.974     348522.074     348523.477          0.654
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        247.729        247.727        247.728          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/30/2020 02:35:04 --

mdtest-hard-write
-- started at 10/30/2020 01:43:14 --

mdtest-3.4.0+dev was launched with 756 total task(s) on 54 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/client/io500/2020.10.30-00.59.02/mdtest-hard' '-x' './results/2020.10.30-00.59.02/mdtest-hard.stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /lustre/client/io500/2020.10.30-00.59.02
FS: 0.0 GiB   Used FS: -nan%   Inodes: -106193898.1 Mi   Used Inodes: 100.0%

Nodemap: 111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2216 Shifting ranks by 14 for each phase.
756 tasks, 756000000 files
Continue stonewall hit min: 93328 max: 114205 avg: 103452.9 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      62724.902      62724.384      62724.636          0.116
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     260698.726             NA
   Tree creation             :          4.648          4.648          4.648          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       1376.482       1376.471       1376.476          0.003
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.003             NA
   Tree creation             :          0.215          0.215          0.215          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 10/30/2020 02:07:39 --

result
version         = io500-sc20_v3
config-hash     = 5876522F
result-dir      = ./results/2020.10.30-00.59.02
; START 2020-10-30 01:00:48


[ior-easy-write]
t_start         = 2020-10-30 01:00:48
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-easy.stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
throughput-stonewall = 57117.49
score           = 98.302897
t_delta         = 423.2879
t_end           = 2020-10-30 01:07:52

[mdtest-easy-write]
t_start         = 2020-10-30 01:07:52
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-easy -x ./results/2020.10.30-00.59.02/mdtest-easy.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 1375.456403
score           = 319.521106
t_delta         = 1701.0213
t_end           = 2020-10-30 01:36:17

[timestamp]
t_start         = 2020-10-30 01:36:17
t_delta         = 0.0008
t_end           = 2020-10-30 01:36:17

[ior-hard-write]
t_start         = 2020-10-30 01:36:17
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 250000 -w -D 300 -a MPIIO
throughput-stonewall = 13007.06
score           = 19.940947
t_delta         = 415.9450
t_end           = 2020-10-30 01:43:14

[mdtest-hard-write]
t_start         = 2020-10-30 01:43:14
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-hard -x ./results/2020.10.30-00.59.02/mdtest-hard.stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 260.698726
score           = 62.724894
t_delta         = 1462.6176
t_end           = 2020-10-30 02:07:40

[find]
t_start         = 2020-10-30 02:07:40
exe             = ./pfind /lustre/client/io500/2020.10.30-00.59.02 -newer ./results/2020.10.30-00.59.02/timestampfile -size 3901c -name *01* -C -q 10000
found           = 4058334
total-files     = 607503457
score           = 1806.571011
t_delta         = 336.2981
t_end           = 2020-10-30 02:13:17

[ior-easy-read]
t_start         = 2020-10-30 02:13:17
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-easy/ior_file_easy -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-easy.stonewall -t 2m -b 9920000m -F -r -R -a POSIX --posix.odirect
score           = 122.731960
t_delta         = 339.0060
t_end           = 2020-10-30 02:18:57

[mdtest-easy-stat]
t_start         = 2020-10-30 02:18:57
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-easy -x ./results/2020.10.30-00.59.02/mdtest-easy.stonewall -T -a POSIX
score           = 2171.375248
t_delta         = 240.0265
t_end           = 2020-10-30 02:22:58

[ior-hard-read]
t_start         = 2020-10-30 02:22:58
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /lustre/client/io500/2020.10.30-00.59.02/ior-hard/file -O stoneWallingStatusFile=./results/2020.10.30-00.59.02/ior-hard.stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 250000 -r -R -a MPIIO
score           = 17.519461
t_delta         = 476.8069
t_end           = 2020-10-30 02:30:56

[mdtest-hard-stat]
t_start         = 2020-10-30 02:30:56
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-hard -x ./results/2020.10.30-00.59.02/mdtest-hard.stonewall -T -a POSIX
score           = 348.524972
t_delta         = 247.9152
t_end           = 2020-10-30 02:35:04

[mdtest-easy-delete]
t_start         = 2020-10-30 02:35:04
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-easy -x ./results/2020.10.30-00.59.02/mdtest-easy.stonewall -r -a POSIX
score           = 526.144738
t_delta         = 992.6695
t_end           = 2020-10-30 02:51:39

[mdtest-hard-read]
t_start         = 2020-10-30 02:51:39
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-hard -x ./results/2020.10.30-00.59.02/mdtest-hard.stonewall -E -X -a POSIX
score           = 250.473274
t_delta         = 344.7165
t_end           = 2020-10-30 02:57:25

[mdtest-hard-delete]
t_start         = 2020-10-30 02:57:25
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/client/io500/2020.10.30-00.59.02/mdtest-hard -x ./results/2020.10.30-00.59.02/mdtest-hard.stonewall -r -a POSIX
score           = 59.036743
t_delta         = 1465.5956
t_end           = 2020-10-30 03:21:54

[SCORE]
MD              = 347.611424
BW              = 45.310372
SCORE           = 125.500609 
hash            = 70F799B4
; END 2020-10-30 03:23:14
result_summary
IO500 version io500-sc20_v3
[RESULT]       ior-easy-write       98.302897 GiB/s : time 423.288 seconds
[RESULT]    mdtest-easy-write      319.521106 kIOPS : time 1701.021 seconds
[RESULT]       ior-hard-write       19.940947 GiB/s : time 415.945 seconds
[RESULT]    mdtest-hard-write       62.724894 kIOPS : time 1462.618 seconds
[RESULT]                 find     1806.571011 kIOPS : time 336.298 seconds
[RESULT]        ior-easy-read      122.731960 GiB/s : time 339.006 seconds
[RESULT]     mdtest-easy-stat     2171.375248 kIOPS : time 240.027 seconds
[RESULT]        ior-hard-read       17.519461 GiB/s : time 476.807 seconds
[RESULT]     mdtest-hard-stat      348.524972 kIOPS : time 247.915 seconds
[RESULT]   mdtest-easy-delete      526.144738 kIOPS : time 992.669 seconds
[RESULT]     mdtest-hard-read      250.473274 kIOPS : time 344.717 seconds
[RESULT]   mdtest-hard-delete       59.036743 kIOPS : time 1465.596 seconds
[SCORE] Bandwidth 45.310372 GiB/s : IOPS 347.611424 kiops : TOTAL 125.500609