NCSA Industry System

Institution National Center for Supercomputing Applications
Client Procs Per Node
Client Operating System RHEL
Client Operating System Version 7.8
Client Kernel Version 3.10.0-1127.10.1.el7.x86_64

DATA SERVER

Storage Type HDD
Volatile Memory 128
Storage Interface Infiniband
Network Infiniband
Software Version 5.0.5.0
OS Version 7.8

INFORMATION

Client Nodes 8
Client Total Procs 96
Metadata Nodes 4
Metadata Storage Devices 4
Data Nodes 4
Data Storage Devices 84

METADATA

Easy Write 81.88 kIOP/s
Easy Stat 222.30 kIOP/s
Easy Delete 58.88 kIOP/s
Hard Write 8.36 kIOP/s
Hard Read 21.41 kIOP/s
Hard Stat 23.26 kIOP/s
Hard Delete 8.44 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  2 13:08:10 2020
Command line        : ./ior -v -C -Q 1 -g -G 271 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
Machine             : Linux mg005
Start time skew across all tasks: 31.58 sec
TestID              : 0
StartTime           : Thu Jul  2 13:08:10 2020
Path                : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy
FS                  : 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 30.7%
Participating tasks: 96
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 8
tasks               : 96
clients per node    : 12
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 908.20 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Thu Jul  2 13:08:10 2020

WARNING: Expected aggregate file size       = 998579896320000.
WARNING: Stat() of aggregate file size      = 6976771719168.
WARNING: Using actual aggregate bytes moved = 6976771719168.
read      10318      2579.64    0.000260    10158080000 4096       0.009528   644.81     0.000572   644.82     0   
Max Read:  10318.42 MiB/sec (10819.64 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        10318.42   10318.42   10318.42       0.00    2579.60    2579.60    2579.60       0.00  644.82448         NA            NA     0     96  12    1   1     1        1         0    0      1 10401873920000  4194304 6653568.0 POSIX      0
Finished            : Thu Jul  2 13:18:55 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  2 12:34:46 2020
Command line        : ./ior -v -C -Q 1 -g -G 271 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
Machine             : Linux mg005
Start time skew across all tasks: 31.58 sec
TestID              : 0
StartTime           : Thu Jul  2 12:34:46 2020
Path                : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy
FS                  : 3659.2 TiB   Used FS: 44.5%   Inodes: 266.5 Mi   Used Inodes: 13.7%
Participating tasks: 96
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 8
tasks               : 96
clients per node    : 12
repetitions         : 1
xfersize            : 4 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 908.20 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Thu Jul  2 12:34:46 2020
0: stonewalling pairs accessed: 5766
stonewalling pairs accessed min: 5610 max: 17327 -- min data: 21.9 GiB mean data: 38.7 GiB time: 300.2s
WARNING: Expected aggregate file size       = 998579896320000.
WARNING: Stat() of aggregate file size      = 6976771719168.
WARNING: Using actual aggregate bytes moved = 6976771719168.
WARNING: maybe caused by deadlineForStonewalling
write     9601       2400.57    0.000279    10158080000 4096       0.120127   692.92     0.001550   693.04     0   
Max Write: 9600.59 MiB/sec (10066.95 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        9600.59    9600.59    9600.59       0.00    2400.15    2400.15    2400.15       0.00  693.03725     300.19      12677.48     0     96  12    1   1     1        1         0    0      1 10401873920000  4194304 6653568.0 POSIX      0
Finished            : Thu Jul  2 12:46:19 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  2 13:22:24 2020
Command line        : ./ior -v -C -Q 1 -g -G 27 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/file -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
Machine             : Linux mg005
Start time skew across all tasks: 31.58 sec
TestID              : 0
StartTime           : Thu Jul  2 13:22:24 2020
Path                : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard
FS                  : 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 30.7%
Participating tasks: 96
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 8
tasks               : 96
clients per node    : 12
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 41.04 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Thu Jul  2 13:22:24 2020

0: stonewalling pairs accessed: 55094
stonewalling pairs accessed min: 55094 max: 55094 -- min data: 2.4 GiB mean data: 2.4 GiB time: 200.6s
WARNING: Expected aggregate file size       = 45127680000000.
WARNING: Stat() of aggregate file size      = 248626440192.
WARNING: Using actual aggregate bytes moved = 248626440192.
read      1181.67    26359      200.65      45.91      45.91      0.002561   200.65     0.000738   200.65     0   
Max Read:  1181.67 MiB/sec (1239.08 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         1181.67    1181.67    1181.67       0.00   26358.82   26358.82   26358.82       0.00  200.65483     200.61       1181.95     0     96  12    1   0     1        1         0    0 10000000    47008    47008  237108.7 POSIX      0
Finished            : Thu Jul  2 13:25:44 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  2 12:55:46 2020
Command line        : ./ior -v -C -Q 1 -g -G 27 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/file -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
Machine             : Linux mg005
Start time skew across all tasks: 31.58 sec
TestID              : 0
StartTime           : Thu Jul  2 12:55:46 2020
Path                : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard
FS                  : 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 29.7%
Participating tasks: 96
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 8
tasks               : 96
clients per node    : 12
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 41.04 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Thu Jul  2 12:55:46 2020
0: stonewalling pairs accessed: 55008
stonewalling pairs accessed min: 55008 max: 55094 -- min data: 2.4 GiB mean data: 2.4 GiB time: 300.0s
WARNING: Expected aggregate file size       = 45127680000000.
WARNING: Stat() of aggregate file size      = 248626440192.
WARNING: Using actual aggregate bytes moved = 248626440192.
WARNING: maybe caused by deadlineForStonewalling
write     789.01     17602      300.46      45.91      45.91      0.021906   300.48     0.013278   300.51     0   
Max Write: 789.01 MiB/sec (827.34 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write         789.01     789.01     789.01       0.00   17599.96   17599.96   17599.96       0.00  300.51335     300.03        789.69     0     96  12    1   0     1        1         0    0 10000000    47008    47008  237108.7 POSIX      0
Finished            : Thu Jul  2 13:00:47 2020
mdtest-easy-delete
-- started at 07/02/2020 13:28:08 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy-stonewall' '-r' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 30.7%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 96000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      58882.519      58882.503      58882.511          0.004
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.429          0.429          0.429          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        787.196        787.196        787.196          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          2.329          2.329          2.329          0.000
-- finished at 07/02/2020 13:41:17 --

mdtest-easy-stat
-- started at 07/02/2020 13:18:55 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy-stonewall' '-T' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 30.7%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 96000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     222295.068     222294.923     222294.995          0.033
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        208.516        208.516        208.516          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/02/2020 13:22:24 --

mdtest-easy-write
-- started at 07/02/2020 12:46:19 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '1000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 13.7%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 96000000 files
Continue stonewall hit min: 251861 max: 482834 avg: 376163.9 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      81877.750      81877.675      81877.721          0.019
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     120614.383             NA
   Tree creation             :          1.601          1.601          1.601          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        566.114        566.113        566.113          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.398             NA
   Tree creation             :          0.624          0.624          0.624          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/02/2020 12:55:46 --

mdtest-hard-delete
-- started at 07/02/2020 13:43:53 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '600000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall' '-r' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 15.7%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 57600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :       8439.443       8439.440       8439.441          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          4.563          4.563          4.563          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        395.139        395.139        395.139          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.219          0.219          0.219          0.000
-- finished at 07/02/2020 13:50:28 --

mdtest-hard-read
-- started at 07/02/2020 13:41:17 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '600000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 15.9%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 57600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :      21407.321      21407.303      21407.311          0.004
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        155.776        155.776        155.776          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/02/2020 13:43:53 --

mdtest-hard-stat
-- started at 07/02/2020 13:25:44 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '600000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall' '-T' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 30.7%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 57600000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :      23261.788      23261.771      23261.783          0.004
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        143.358        143.358        143.358          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/02/2020 13:28:08 --

mdtest-hard-write
-- started at 07/02/2020 13:00:47 --

mdtest-3.3.0+dev was launched with 96 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '600000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard' '-x' '/magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'POSIX'
Path: /magnus/io500/datafiles/2020.07.02-12.32.35-app
FS: 3659.2 TiB   Used FS: 44.7%   Inodes: 266.5 Mi   Used Inodes: 29.7%

Nodemap: 111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 12 for each phase.
96 tasks, 57600000 files
Continue stonewall hit min: 24384 max: 34737 avg: 27219.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :       8360.333       8360.326       8360.329          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA       8709.706             NA
   Tree creation             :       3898.240       3898.240       3898.240          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        398.878        398.878        398.878          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.022             NA
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/02/2020 13:07:26 --

result
version         = io500-isc20_v4-1-g18c6d03b40b0
config-hash     = FD3D34E1
result-dir      = ./results/2020.07.02-12.32.35-app
; START 2020-07-02 12:34:46


[ior-easy-write]
t_start         = 2020-07-02 12:34:46
exe             = ./ior -v -C -Q 1 -g -G 271 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/stonewall -t 4m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a POSIX
throughput-stonewall = 514.81
score           = 9.375578
t_delta         = 693.1188
t_end           = 2020-07-02 12:46:19

[mdtest-easy-write]
t_start         = 2020-07-02 12:46:19
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 120.614383
score           = 81.877707
t_delta         = 567.2596
t_end           = 2020-07-02 12:55:46

[timestamp]
t_start         = 2020-07-02 12:55:46
t_delta         = 0.0007
t_end           = 2020-07-02 12:55:46

[ior-hard-write]
t_start         = 2020-07-02 12:55:46
exe             = ./ior -v -C -Q 1 -g -G 27 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/file -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a POSIX
throughput-stonewall = 73.91
score           = 0.770520
t_delta         = 300.5216
t_end           = 2020-07-02 13:00:47

[mdtest-hard-write]
t_start         = 2020-07-02 13:00:47
exe             = ./mdtest -n 600000 -t -w 3901 -e 3901 -P -N 1 -F -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall -C -Y -W 300 -a POSIX
rate-stonewall  = 8.709706
score           = 8.360328
t_delta         = 399.3945
t_end           = 2020-07-02 13:07:26

[find]
t_start         = 2020-07-02 13:07:26
exe             = ./pfind /magnus/io500/datafiles/2020.07.02-12.32.35-app -newer /magnus/io500/datafiles/2020.07.02-12.32.35-app/timestampfile -size 3901c -name *01* -C -q 10000
nproc           = 96
found           = 94464
total-files     = 49686918
score           = 1130.609562
t_delta         = 44.1302
t_end           = 2020-07-02 13:08:10

[ior-easy-read]
t_start         = 2020-07-02 13:08:10
exe             = ./ior -v -C -Q 1 -g -G 271 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-easy/stonewall -t 4m -b 9920000m -F -r -R -a POSIX
score           = 10.076579
t_delta         = 644.8307
t_end           = 2020-07-02 13:18:55

[mdtest-easy-stat]
t_start         = 2020-07-02 13:18:55
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy-stonewall -T -a POSIX
score           = 222.295014
t_delta         = 208.5195
t_end           = 2020-07-02 13:22:24

[ior-hard-read]
t_start         = 2020-07-02 13:22:24
exe             = ./ior -v -C -Q 1 -g -G 27 -k -e -o /magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/file -O stoneWallingStatusFile=/magnus/io500/datafiles/2020.07.02-12.32.35-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a POSIX
score           = 1.153979
t_delta         = 200.6580
t_end           = 2020-07-02 13:25:44

[mdtest-hard-stat]
t_start         = 2020-07-02 13:25:44
exe             = ./mdtest -n 600000 -t -w 3901 -e 3901 -P -N 1 -F -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall -T -a POSIX
score           = 23.261785
t_delta         = 143.3606
t_end           = 2020-07-02 13:28:08

[mdtest-easy-delete]
t_start         = 2020-07-02 13:28:08
exe             = ./mdtest -n 1000000 -u -L -F -P -N 1 -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-easy-stonewall -r -a POSIX
score           = 58.882503
t_delta         = 789.5532
t_end           = 2020-07-02 13:41:17

[mdtest-hard-read]
t_start         = 2020-07-02 13:41:17
exe             = ./mdtest -n 600000 -t -w 3901 -e 3901 -P -N 1 -F -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall -E -X -a POSIX
score           = 21.407310
t_delta         = 155.7833
t_end           = 2020-07-02 13:43:53

[mdtest-hard-delete]
t_start         = 2020-07-02 13:43:53
exe             = ./mdtest -n 600000 -t -w 3901 -e 3901 -P -N 1 -F -d /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard -x /magnus/io500/datafiles/2020.07.02-12.32.35-app/mdtest-hard-stonewall -r -a POSIX
score           = 8.439440
t_delta         = 395.3641
t_end           = 2020-07-02 13:50:28

[SCORE]
MD              = 50.540793
BW              = 3.027423
SCORE           = 12.369656 
hash            = AC4C1A3B
; END 2020-07-02 13:50:29
result_summary
IO500 version io500-isc20_v4-1-g18c6d03b40b0
[RESULT]       ior-easy-write        9.375578 GiB/s  : time 693.119 seconds
[RESULT]    mdtest-easy-write       81.877707 kIOPS : time 567.260 seconds
[RESULT]       ior-hard-write        0.770520 GiB/s  : time 300.522 seconds
[RESULT]    mdtest-hard-write        8.360328 kIOPS : time 399.394 seconds
[RESULT]                 find     1130.609562 kIOPS : time 44.130 seconds
[RESULT]        ior-easy-read       10.076579 GiB/s  : time 644.831 seconds
[RESULT]     mdtest-easy-stat      222.295014 kIOPS : time 208.520 seconds
[RESULT]        ior-hard-read        1.153979 GiB/s  : time 200.658 seconds
[RESULT]     mdtest-hard-stat       23.261785 kIOPS : time 143.361 seconds
[RESULT]   mdtest-easy-delete       58.882503 kIOPS : time 789.553 seconds
[RESULT]     mdtest-hard-read       21.407310 kIOPS : time 155.783 seconds
[RESULT]   mdtest-hard-delete        8.439440 kIOPS : time 395.364 seconds
[SCORE] Bandwidth 3.027423 GB/s : IOPS 50.540793 kiops : TOTAL 12.369656