Frontera

Institution TACC
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8
Client Kernel Version 3.10.0-1127.13.1.el7.x86_64

DATA SERVER

Storage Type Optane Persistent Memory
Volatile Memory 256GB
Storage Interface DDR-T
Network Infiniband
Software Version 1.1
OS Version 7.8

INFORMATION

Client Nodes 10
Client Total Procs 420
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 15
Data Storage Devices 24

METADATA

Easy Write 4,835.24 kIOP/s
Easy Stat 4,653.56 kIOP/s
Easy Delete 2,787.23 kIOP/s
Hard Write 2,323.15 kIOP/s
Hard Read 3,206.32 kIOP/s
Hard Stat 3,971.63 kIOP/s
Hard Delete 2,710.68 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 16:47:53 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/stonewall -t 256k -b 1000000m -r -R -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-001.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Sat Jul 11 16:47:56 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 420
tasks               : 420
clients per node    : 1
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 976.56 GiB
aggregate filesize  : 400.54 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 440401920000000.
WARNING: Stat() of aggregate file size      = 439444025376768.
WARNING: Using actual aggregate bytes moved = 38086178242560.
read      72350      291097     0.000125    1024000000 256.00     2.92       499.10     0.014455   502.03     0   
Max Read:  72349.66 MiB/sec (75864.12 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        72349.66   72349.66   72349.66       0.00  289398.64  289398.64  289398.64       0.00  502.03153         NA            NA     0    420   1    1   0     1        1         0    0      1 1048576000000   262144 36321808.0 DFS      0
Finished            : Sat Jul 11 16:56:21 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 16:19:10 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/stonewall -t 256k -b 1000000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-001.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Sat Jul 11 16:19:13 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 420
tasks               : 420
clients per node    : 1
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 976.56 GiB
aggregate filesize  : 400.54 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 278082 max: 345922 -- min data: 67.9 GiB mean data: 73.3 GiB time: 300.0s
WARNING: Expected aggregate file size       = 440401920000000.
WARNING: Stat() of aggregate file size      = 439444025376768.
WARNING: Using actual aggregate bytes moved = 38086178242560.
WARNING: maybe caused by deadlineForStonewalling
write     99102      402302     0.000075    1024000000 256.00     5.37       361.14     0.001468   366.51     0   
Max Write: 99101.91 MiB/sec (103915.88 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       99101.91   99101.91   99101.91       0.00  396407.62  396407.62  396407.62       0.00  366.50970     300.02     105121.29     0    420   1    1   0     1        1         0    0      1 1048576000000   262144 36321808.0 DFS      0
Finished            : Sat Jul 11 16:25:23 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 17:01:53 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.chunk_size=1880320 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-001.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Sat Jul 11 17:01:55 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 420
tasks               : 420
clients per node    : 1
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 179.56 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1701801 max: 1701801 -- min data: 74.5 GiB mean data: 74.5 GiB time: 464.0s
WARNING: Expected aggregate file size       = 197433600000000.
WARNING: Stat() of aggregate file size      = 33599269791360.
WARNING: Using actual aggregate bytes moved = 33599269791360.
read      68643      1540239    464.05      45.91      45.91      2.73       464.06     0.017214   466.80     0   
Max Read:  68643.15 MiB/sec (71977.56 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        68643.15   68643.15   68643.15       0.00 1531176.84 1531176.84 1531176.84       0.00  466.80201     463.96      69063.34     0    420   1    1   0     1        1         0    0 10000000    47008    47008 32042760.0 DFS      0
Finished            : Sat Jul 11 17:09:45 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 16:30:45 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.chunk_size=1880320 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-001.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Sat Jul 11 16:30:48 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 420
tasks               : 420
clients per node    : 1
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 179.56 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1327923 max: 1701801 -- min data: 58.1 GiB mean data: 59.4 GiB time: 300.1s
WARNING: Expected aggregate file size       = 197433600000000.
WARNING: Stat() of aggregate file size      = 33599269791360.
WARNING: Using actual aggregate bytes moved = 33599269791360.
WARNING: maybe caused by deadlineForStonewalling
write     84711      1915895    300.12      45.91      45.91      5.19       373.07     0.002981   378.26     0   
Max Write: 84710.53 MiB/sec (88825.43 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       84710.53   84710.53   84710.53       0.00 1889581.12 1889581.12 1889581.12       0.00  378.26183     300.05      85081.32     0    420   1    1   0     1        1         0    0 10000000    47008    47008 32042760.0 DFS      0
Finished            : Sat Jul 11 16:37:10 2020
mdtest-easy-delete
-- started at 07/11/2020 17:12:59 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy-stonewall' '-r' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    2787256.125    2787231.767    2787236.523          3.791
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.107          0.107          0.107          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        542.485        542.481        542.484          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          9.367          9.367          9.367          0.000
-- finished at 07/11/2020 17:22:11 --

mdtest-easy-stat
-- started at 07/11/2020 16:56:25 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy-stonewall' '-T' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    4653583.168    4653558.063    4653561.295          2.748
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        324.920        324.918        324.919          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 17:01:50 --

mdtest-easy-write
-- started at 07/11/2020 16:25:26 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files
Continue stonewall hit min: 3381163 max: 3600077 avg: 3548661.8 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    4835288.072    4835236.122    4835246.096          5.253
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    5019235.354             NA
   Tree creation             :          0.327          0.327          0.327          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        312.711        312.708        312.711          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        296.945             NA
   Tree creation             :          3.054          3.054          3.054          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 16:30:42 --

mdtest-hard-delete
-- started at 07/11/2020 17:26:11 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall' '-r' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    2710716.095    2710676.804    2710682.481          7.175
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.001          0.001          0.001          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        269.233        269.229        269.232          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :       1521.599       1521.599       1521.599          0.000
-- finished at 07/11/2020 17:56:02 --

mdtest-hard-read
-- started at 07/11/2020 17:22:17 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :    3206346.410    3206310.692    3206320.879          9.870
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        227.615        227.612        227.614          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 17:26:05 --

mdtest-hard-stat
-- started at 07/11/2020 17:09:49 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall' '-T' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    3971690.444    3971610.158    3971635.402         15.509
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        183.755        183.751        183.754          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 17:12:53 --

mdtest-hard-write
-- started at 07/11/2020 16:37:13 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 420 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec' '--dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files
Continue stonewall hit min: 1645380 max: 1737628 avg: 1712237.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    2323182.657    2323149.032    2323153.816          5.121
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    2397258.647             NA
   Tree creation             :        612.486        612.486        612.486          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        314.144        314.140        314.144          0.001
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.984             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 16:42:28 --

result
version         = io500-isc20-38
config-hash     = 74F1BADF
result-dir      = /home1/06758/mschaara/io-500/results/2020.07.11-14.40.08-app
; START 2020-07-11 16:19:10


[ior-easy-write]
t_start         = 2020-07-11 16:19:10
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/stonewall -t 256k -b 1000000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
throughput-stonewall = 35293.93
score           = 96.779205
t_delta         = 372.5593
t_end           = 2020-07-11 16:25:23

[mdtest-easy-write]
t_start         = 2020-07-11 16:25:23
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy-stonewall -C -Y -W 300 -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
rate-stonewall  = 5019.235354
score           = 4835.237478
t_delta         = 321.1970
t_end           = 2020-07-11 16:30:44

[timestamp]
t_start         = 2020-07-11 16:30:45
t_delta         = 0.5936
t_end           = 2020-07-11 16:30:45

[ior-hard-write]
t_start         = 2020-07-11 16:30:45
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.chunk_size=1880320 --dfs.prefix=/tmp/dfuse
throughput-stonewall = 27681.62
score           = 82.725127
t_delta         = 384.1226
t_end           = 2020-07-11 16:37:10

[mdtest-hard-write]
t_start         = 2020-07-11 16:37:10
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall -C -Y -W 300 -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
rate-stonewall  = 2397.258647
score           = 2323.149744
t_delta         = 319.7396
t_end           = 2020-07-11 16:42:30

[find]
t_start         = 2020-07-11 16:42:30
exe             = mpirun -np 420 --hostfile /home1/06758/mschaara/io-500/config/cli_hosts_10 /home1/06758/mschaara/io-500/install/mfu/bin/dfind --pool 372e1597-1708-4f1e-aa78-1cceb4c7c6ec --cont af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --svcl 1 --prefix /tmp/dfuse --stonewall 300 /tmp/dfuse/datafiles/2020.07.11-14.40.08-app -newer /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/timestampfile -size 3901 -name "*01*" -o /home1/06758/mschaara/io-500/results/2020.07.11-14.40.08-app/find_output.txt
nproc           = 1
last-output     = "MATCHED 174590/852547580"
total-files     = 852547580
found           = 174590
score           = 2646.562830
t_delta         = 322.6679
t_end           = 2020-07-11 16:47:53

[ior-easy-read]
t_start         = 2020-07-11 16:47:53
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-easy/stonewall -t 256k -b 1000000m -r -R -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
score           = 70.653964
t_delta         = 508.0449
t_end           = 2020-07-11 16:56:21

[mdtest-easy-stat]
t_start         = 2020-07-11 16:56:22
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy-stonewall -T -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 4653.560522
t_delta         = 330.7109
t_end           = 2020-07-11 17:01:52

[ior-hard-read]
t_start         = 2020-07-11 17:01:53
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.11-14.40.08-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.chunk_size=1880320 --dfs.prefix=/tmp/dfuse
score           = 67.034327
t_delta         = 472.7202
t_end           = 2020-07-11 17:09:45

[mdtest-hard-stat]
t_start         = 2020-07-11 17:09:46
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall -T -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 3971.627163
t_delta         = 189.3360
t_end           = 2020-07-11 17:12:55

[mdtest-easy-delete]
t_start         = 2020-07-11 17:12:56
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-easy-stonewall -r -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 2787.233994
t_delta         = 557.3487
t_end           = 2020-07-11 17:22:13

[mdtest-hard-read]
t_start         = 2020-07-11 17:22:14
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall -E -X -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 3206.315125
t_delta         = 233.6677
t_end           = 2020-07-11 17:26:07

[mdtest-hard-delete]
t_start         = 2020-07-11 17:26:08
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.11-14.40.08-app/mdtest-hard-stonewall -r -a DFS --dfs.pool=372e1597-1708-4f1e-aa78-1cceb4c7c6ec --dfs.cont=af091f54-7e7d-4c6b-bc70-a5d7d16e4955 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 2710.679267
t_delta         = 1797.2978
t_end           = 2020-07-11 17:56:05

[SCORE]
MD              = 3277.355885
BW              = 78.471747
SCORE           = 507.129018 
hash            = A319338D
; END 2020-07-11 17:56:06
result_summary
IO500 version io500-isc20-38
[RESULT]       ior-easy-write       96.779205 GiB/s  : time 372.559 seconds
[RESULT]    mdtest-easy-write     4835.237478 kIOPS : time 321.197 seconds
[RESULT]       ior-hard-write       82.725127 GiB/s  : time 384.123 seconds
[RESULT]    mdtest-hard-write     2323.149744 kIOPS : time 319.740 seconds
[RESULT]                 find     2646.562830 kIOPS : time 322.668 seconds
[RESULT]        ior-easy-read       70.653964 GiB/s  : time 508.045 seconds
[RESULT]     mdtest-easy-stat     4653.560522 kIOPS : time 330.711 seconds
[RESULT]        ior-hard-read       67.034327 GiB/s  : time 472.720 seconds
[RESULT]     mdtest-hard-stat     3971.627163 kIOPS : time 189.336 seconds
[RESULT]   mdtest-easy-delete     2787.233994 kIOPS : time 557.349 seconds
[RESULT]     mdtest-hard-read     3206.315125 kIOPS : time 233.668 seconds
[RESULT]   mdtest-hard-delete     2710.679267 kIOPS : time 1797.298 seconds
[SCORE] Bandwidth 78.471747 GB/s : IOPS 3277.355885 kiops : TOTAL 507.129018