Polytechnic RSC Tornado

Institution SPbPU
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7.1908
Client Kernel Version 3.10.0-1062.1.1.el7.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 187GB
Storage Interface NVMe over Fabric
Network IB
Software Version 2.13.0
OS Version 7.7.1908

INFORMATION

Client Nodes 59
Client Total Procs 944
Metadata Nodes 4
Metadata Storage Devices 2
Data Nodes 55
Data Storage Devices 109

METADATA

Easy Write 188.81 kIOP/s
Easy Stat 735.71 kIOP/s
Easy Delete 134.52 kIOP/s
Hard Write 88.86 kIOP/s
Hard Read 304.47 kIOP/s
Hard Stat 380.50 kIOP/s
Hard Delete 64.77 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 06:32:41 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX --posix.odirect
Machine             : Linux n06p001
TestID              : 0
StartTime           : Sat Jul 11 06:32:41 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy
FS                  : 196.3 TiB   Used FS: 24.1%   Inodes: 917.3 Mi   Used Inodes: 20.4%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 8930.66 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 9819368980480000.
WARNING: Stat() of aggregate file size      = 50569750249472.
WARNING: Using actual aggregate bytes moved = 50569750249472.
read      199648     99940      0.000049    10158080000 2048.00    0.133158   241.28     0.258027   241.56     0   
Max Read:  199647.53 MiB/sec (209345.61 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       199647.53  199647.53  199647.53       0.00   99823.77   99823.77   99823.77       0.00  241.56107         NA            NA     0    944  16    1   1     1        1         0    0      1 10401873920000  2097152 48227072.0 POSIX      0
Finished            : Sat Jul 11 06:36:43 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 05:26:59 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
Machine             : Linux n06p001
TestID              : 0
StartTime           : Sat Jul 11 05:26:59 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy
FS                  : 196.3 TiB   Used FS: 0.0%   Inodes: 793.2 Mi   Used Inodes: 0.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 8930.66 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 20200 max: 25544 -- min data: 39.5 GiB mean data: 42.6 GiB time: 300.1s
WARNING: Expected aggregate file size       = 9819368980480000.
WARNING: Stat() of aggregate file size      = 50569750249472.
WARNING: Using actual aggregate bytes moved = 50569750249472.
WARNING: maybe caused by deadlineForStonewalling
write     127792     63915      0.000073    10158080000 2048.00    0.076703   377.28     0.054066   377.39     0   
Max Write: 127792.31 MiB/sec (133999.95 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      127792.31  127792.31  127792.31       0.00   63896.16   63896.16   63896.16       0.00  377.38633     300.13     137049.27     0    944  16    1   1     1        1         0    0      1 10401873920000  2097152 48227072.0 POSIX      0
Finished            : Sat Jul 11 05:33:17 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 06:39:44 2020
Command line        : ./ior -v -C -Q 1 -g -G 27 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/file -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 34000 -r -R -a MPIIO
Machine             : Linux n06p001
Start time skew across all tasks: 983255.31 sec
TestID              : 0
StartTime           : Sat Jul 11 06:39:44 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard
FS                  : 196.3 TiB   Used FS: 24.1%   Inodes: 917.3 Mi   Used Inodes: 20.4%
Participating tasks: 944
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 34000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 1.37 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Sat Jul 11 06:39:45 2020

0: stonewalling pairs accessed: 25735
stonewalling pairs accessed min: 25735 max: 25735 -- min data: 1.1 GiB mean data: 1.1 GiB time: 355.1s
WARNING: Expected aggregate file size       = 1508768768000.
WARNING: Stat() of aggregate file size      = 1142004830720.
WARNING: Using actual aggregate bytes moved = 1142004830720.
read      3053.79    68357      355.21      45.91      45.91      1.17       355.40     0.223332   356.64     0   
Max Read:  3053.79 MiB/sec (3202.13 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         3053.79    3053.79    3053.79       0.00   68118.92   68118.92   68118.92       0.00  356.63866     355.10       3067.01     0    944  16    1   0     1        1         0    0  34000    47008    47008 1089100.6 MPIIO      0
Finished            : Sat Jul 11 06:45:42 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 05:47:44 2020
Command line        : ./ior -v -C -Q 1 -g -G 27 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/file -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 34000 -w -D 300 -a MPIIO
Machine             : Linux n06p001
Start time skew across all tasks: 983255.35 sec
TestID              : 0
StartTime           : Sat Jul 11 05:47:44 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard
FS                  : 196.3 TiB   Used FS: 23.4%   Inodes: 917.3 Mi   Used Inodes: 13.5%
Participating tasks: 944
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 34000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 1.37 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Sat Jul 11 05:48:01 2020
0: stonewalling pairs accessed: 15275
stonewalling pairs accessed min: 15264 max: 25735 -- min data: 0.7 GiB mean data: 1.0 GiB time: 300.1s
WARNING: Expected aggregate file size       = 1508768768000.
WARNING: Stat() of aggregate file size      = 1142004830720.
WARNING: Using actual aggregate bytes moved = 1142004830720.
WARNING: maybe caused by deadlineForStonewalling
write     3049.25    71367      340.38      45.91      45.91      16.67      340.41     0.166796   357.17     0   
Max Write: 3049.25 MiB/sec (3197.37 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        3049.25    3049.25    3049.25       0.00   68017.51   68017.51   68017.51       0.00  357.17041     300.11       3211.57     0    944  16    1   0     1        1         0    0  34000    47008    47008 1089100.6 MPIIO      0
Finished            : Sat Jul 11 05:53:42 2020
mdtest-easy-delete
-- started at 07/11/2020 06:48:39 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 196.3 TiB   Used FS: 24.1%   Inodes: 917.3 Mi   Used Inodes: 20.4%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     134523.629     134513.629     134518.951          2.076
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.059          1.059          1.059          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        967.225        967.153        967.186          0.015
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.945          0.945          0.945          0.000
-- finished at 07/11/2020 07:04:48 --

mdtest-easy-stat
-- started at 07/11/2020 06:36:45 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 196.3 TiB   Used FS: 24.1%   Inodes: 917.3 Mi   Used Inodes: 20.4%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     735801.898     735457.236     735655.151         72.942
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        176.903        176.821        176.856          0.018
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 06:39:42 --

mdtest-easy-write
-- started at 07/11/2020 05:33:18 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 196.3 TiB   Used FS: 23.4%   Inodes: 793.2 Mi   Used Inodes: 0.0%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files
Continue stonewall hit min: 69167 max: 137823 avg: 104884.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     188818.361     188788.983     188804.700          4.915
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     330216.880             NA
   Tree creation             :          9.896          9.896          9.896          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        689.155        689.048        689.098          0.018
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.836             NA
   Tree creation             :          0.101          0.101          0.101          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 05:47:43 --

mdtest-hard-delete
-- started at 07/11/2020 07:08:29 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 196.3 TiB   Used FS: 24.1%   Inodes: 793.2 Mi   Used Inodes: 7.9%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      64776.168      64770.018      64772.106          1.126
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.164          0.164          0.164          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       1020.444       1020.347       1020.411          0.018
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          6.084          6.084          6.084          0.000
-- finished at 07/11/2020 07:25:37 --

mdtest-hard-read
-- started at 07/11/2020 07:04:50 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 196.3 TiB   Used FS: 24.1%   Inodes: 793.2 Mi   Used Inodes: 7.9%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     304525.709     304389.101     304455.715         28.424
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        217.137        217.040        217.090          0.020
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 07:08:28 --

mdtest-hard-stat
-- started at 07/11/2020 06:45:43 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 196.3 TiB   Used FS: 24.1%   Inodes: 917.3 Mi   Used Inodes: 20.4%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     380567.816     380419.452     380495.859         28.065
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        173.740        173.672        173.705          0.013
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 06:48:38 --

mdtest-hard-write
-- started at 07/11/2020 05:53:43 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: ./mdtest '-n' '1000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app
FS: 0.0 GiB   Used FS: -nan%   Inodes: -113373821.3 Mi   Used Inodes: 100.0%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files
Continue stonewall hit min: 64448 max: 70015 avg: 67479.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      88864.861      88855.102      88860.225          1.702
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     212383.141             NA
   Tree creation             :         12.906         12.906         12.906          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        743.842        743.760        743.799          0.014
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.934             NA
   Tree creation             :          0.077          0.077          0.077          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 06:10:21 --

result
version         = io500-isc20_v3-3-g18c6d03b40b0
config-hash     = 79D0E1C6
result-dir      = ./results/2020.07.11-05.26.49-app
; START 2020-07-11 05:26:57


[ior-easy-write]
t_start         = 2020-07-11 05:26:57
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX --posix.odirect
throughput-stonewall = 100478.66
score           = 124.797180
t_delta         = 380.3455
t_end           = 2020-07-11 05:33:17

[mdtest-easy-write]
t_start         = 2020-07-11 05:33:17
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 330.216880
score           = 188.810311
t_delta         = 865.6456
t_end           = 2020-07-11 05:47:43

[timestamp]
t_start         = 2020-07-11 05:47:43
t_delta         = 0.0026
t_end           = 2020-07-11 05:47:43

[ior-hard-write]
t_start         = 2020-07-11 05:47:43
exe             = ./ior -v -C -Q 1 -g -G 27 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/file -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 34000 -w -D 300 -a MPIIO
throughput-stonewall = 2487.70
score           = 2.977780
t_delta         = 359.1609
t_end           = 2020-07-11 05:53:42

[mdtest-hard-write]
t_start         = 2020-07-11 05:53:42
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 212.383141
score           = 88.857976
t_delta         = 998.7831
t_end           = 2020-07-11 06:10:21

[find]
t_start         = 2020-07-11 06:10:21
exe             = ./pfind /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app -newer /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/timestampfile -size 3901c -name *01* -C -q 10000
found           = 2489850
total-files     = 196200022
score           = 146.540073
t_delta         = 1339.0652
t_end           = 2020-07-11 06:32:40

[ior-easy-read]
t_start         = 2020-07-11 06:32:40
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a POSIX --posix.odirect
score           = 194.968294
t_delta         = 242.9673
t_end           = 2020-07-11 06:36:43

[mdtest-easy-stat]
t_start         = 2020-07-11 06:36:43
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy-stonewall -T -a POSIX
score           = 735.712630
t_delta         = 179.3527
t_end           = 2020-07-11 06:39:42

[ior-hard-read]
t_start         = 2020-07-11 06:39:43
exe             = ./ior -v -C -Q 1 -g -G 27 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/file -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 34000 -r -R -a MPIIO
score           = 2.982220
t_delta         = 359.3232
t_end           = 2020-07-11 06:45:42

[mdtest-hard-stat]
t_start         = 2020-07-11 06:45:42
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall -T -a POSIX
score           = 380.504663
t_delta         = 175.8626
t_end           = 2020-07-11 06:48:38

[mdtest-easy-delete]
t_start         = 2020-07-11 06:48:38
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-easy-stonewall -r -a POSIX
score           = 134.522826
t_delta         = 970.7914
t_end           = 2020-07-11 07:04:48

[mdtest-hard-read]
t_start         = 2020-07-11 07:04:48
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 304.474903
t_delta         = 219.4346
t_end           = 2020-07-11 07:08:28

[mdtest-hard-delete]
t_start         = 2020-07-11 07:08:28
exe             = ./mdtest -n 1000000 -t -w 3901 -e 3901 -P -N 1 -F -d /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard -x /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-app/mdtest-hard-stonewall -r -a POSIX
score           = 64.770776
t_delta         = 1028.8761
t_end           = 2020-07-11 07:25:37

[SCORE]
MD              = 191.727391
BW              = 21.560074
SCORE           = 64.293521 
hash            = EA28C17C
; END 2020-07-11 07:25:39
result_summary
IO500 version io500-isc20_v3-3-g18c6d03b40b0
[RESULT]       ior-easy-write      124.797180 GiB/s  : time 380.345 seconds
[RESULT]    mdtest-easy-write      188.810311 kIOPS : time 865.646 seconds
[RESULT]       ior-hard-write        2.977780 GiB/s  : time 359.161 seconds
[RESULT]    mdtest-hard-write       88.857976 kIOPS : time 998.783 seconds
[RESULT]                 find      146.540073 kIOPS : time 1339.065 seconds
[RESULT]        ior-easy-read      194.968294 GiB/s  : time 242.967 seconds
[RESULT]     mdtest-easy-stat      735.712630 kIOPS : time 179.353 seconds
[RESULT]        ior-hard-read        2.982220 GiB/s  : time 359.323 seconds
[RESULT]     mdtest-hard-stat      380.504663 kIOPS : time 175.863 seconds
[RESULT]   mdtest-easy-delete      134.522826 kIOPS : time 970.791 seconds
[RESULT]     mdtest-hard-read      304.474903 kIOPS : time 219.435 seconds
[RESULT]   mdtest-hard-delete       64.770776 kIOPS : time 1028.876 seconds
[SCORE] Bandwidth 21.560074 GB/s : IOPS 191.727391 kiops : TOTAL 64.293521