Frontera

Institution TACC
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.8
Client Kernel Version 3.10.0-1127.13.1.el7.x86_64

DATA SERVER

Storage Type Optane Persistent Memory
Volatile Memory 256GB
Storage Interface DDR-T
Network Infiniband
Software Version 1.1
OS Version 7.8

INFORMATION

Client Nodes 60
Client Total Procs 1,440
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 16
Data Storage Devices 24

METADATA

Easy Write 11,268.77 kIOP/s
Easy Stat 11,157.56 kIOP/s
Easy Delete 6,438.14 kIOP/s
Hard Write 5,030.24 kIOP/s
Hard Read 7,372.83 kIOP/s
Hard Stat 9,015.77 kIOP/s
Hard Delete 5,908.40 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 00:26:02 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/stonewall -t 256k -b 20000m -r -R -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-011.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Mon Jul 13 00:26:11 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1440
tasks               : 1440
clients per node    : 1
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 19.53 GiB
aggregate filesize  : 27.47 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
read      76461      324255     0.004441    20480000   256.00     21.38      355.28     0.008780   376.66     0   
Max Read:  76461.12 MiB/sec (80175.29 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        76461.12   76461.12   76461.12       0.00  305844.48  305844.48  305844.48       0.00  376.66202         NA            NA     0   1440   1    1   0     1        1         0    0      1 20971520000   262144 28800000.0 DFS      0
Finished            : Mon Jul 13 00:32:30 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Jul 12 23:55:43 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/stonewall -t 256k -b 20000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-011.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Sun Jul 12 23:55:45 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1440
tasks               : 1440
clients per node    : 1
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 19.53 GiB
aggregate filesize  : 27.47 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 75215 max: 80000 -- min data: 18.4 GiB mean data: 19.0 GiB time: 300.0s
write     81885      372440     0.003858    20480000   256.00     42.40      309.31     0.000667   351.71     0   
Max Write: 81884.87 MiB/sec (85862.51 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       81884.87   81884.87   81884.87       0.00  327539.47  327539.47  327539.47       0.00  351.71334     300.00      93171.93     0   1440   1    1   0     1        1         0    0      1 20971520000   262144 28800000.0 DFS      0
Finished            : Mon Jul 13 00:01:40 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 00:37:32 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-011.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Mon Jul 13 00:37:42 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1440
tasks               : 1440
clients per node    : 1
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 615.65 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 567239 max: 567239 -- min data: 24.8 GiB mean data: 24.8 GiB time: 484.3s
WARNING: Expected aggregate file size       = 676915200000000.
WARNING: Stat() of aggregate file size      = 38397270113280.
WARNING: Using actual aggregate bytes moved = 38397270113280.
read      72458      1686575    484.30      45.91      45.91      21.07      484.31     0.004039   505.38     0   
Max Read:  72457.86 MiB/sec (75977.57 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        72457.86   72457.86   72457.86       0.00 1616269.02 1616269.02 1616269.02       0.00  505.37636     484.25      75618.94     0   1440   1    1   0     1        1         0    0 10000000    47008    47008 36618492.0 DFS      0
Finished            : Mon Jul 13 00:46:10 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jul 13 00:07:10 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse
Machine             : Linux c110-011.frontera.tacc.utexas.edu
TestID              : 0
StartTime           : Mon Jul 13 00:07:12 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 1440
tasks               : 1440
clients per node    : 1
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 615.65 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 308581 max: 567239 -- min data: 13.5 GiB mean data: 17.0 GiB time: 300.1s
WARNING: Expected aggregate file size       = 676915200000000.
WARNING: Stat() of aggregate file size      = 38397270113280.
WARNING: Using actual aggregate bytes moved = 38397270113280.
WARNING: maybe caused by deadlineForStonewalling
write     78334      1920325    421.15      45.91      45.91      42.11      425.36     0.003584   467.47     0   
Max Write: 78333.80 MiB/sec (82138.94 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       78333.80   78333.80   78333.80       0.00 1747339.60 1747339.60 1747339.60       0.00  467.46732     300.05      83758.83     0   1440   1    1   0     1        1         0    0 10000000    47008    47008 36618492.0 DFS      0
Finished            : Mon Jul 13 00:15:02 2020
mdtest-easy-delete
-- started at 07/13/2020 00:49:28 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '100000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy-stonewall' '-r' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 144000000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    6438146.374    6438109.066    6438143.077          1.571
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.080          0.080          0.080          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        502.360        502.357        502.358          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         12.429         12.429         12.429          0.000
-- finished at 07/13/2020 00:58:03 --

mdtest-easy-stat
-- started at 07/13/2020 00:32:40 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '100000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy-stonewall' '-T' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 144000000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :   11157573.105   11157549.520   11157563.551          3.135
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        289.871        289.870        289.871          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 00:37:30 --

mdtest-easy-write
-- started at 07/13/2020 00:01:50 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '100000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 144000000000 files
Continue stonewall hit min: 2146118 max: 2246007 avg: 2223617.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :   11268770.812   11268759.682   11268766.719          2.005
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA   11492710.011             NA
   Tree creation             :          0.047          0.047          0.047          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        287.010        287.010        287.010          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        278.612             NA
   Tree creation             :         21.361         21.361         21.361          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 00:06:59 --

mdtest-hard-delete
-- started at 07/13/2020 01:02:02 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall' '-r' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 14400000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    5908397.760    5908390.229    5908394.111          1.681
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        268.444        268.444        268.444          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :       3626.219       3626.219       3626.219          0.000
-- finished at 07/13/2020 02:06:57 --

mdtest-hard-read
-- started at 07/13/2020 00:58:15 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 14400000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :    7372830.862    7372570.521    7372820.895         32.402
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        215.132        215.124        215.124          0.001
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 01:01:50 --

mdtest-hard-stat
-- started at 07/13/2020 00:46:20 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall' '-T' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 14400000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    9015783.499    9015763.460    9015773.211          3.733
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        175.922        175.922        175.922          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 00:49:16 --

mdtest-hard-write
-- started at 07/13/2020 00:15:13 --

mdtest-3.3.0+dev was launched with 1440 total task(s) on 1440 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791' '--dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse'
Nodemap: 100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1440 tasks, 14400000000 files
Continue stonewall hit min: 1025625 max: 1101440 avg: 1067971.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    5030256.683    5030234.732    5030240.482          1.388
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    5126580.477             NA
   Tree creation             :         79.272         79.272         79.272          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        315.308        315.307        315.308          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.981             NA
   Tree creation             :          0.013          0.013          0.013          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/13/2020 00:20:28 --

result
version         = io500-isc20_v4-22
config-hash     = 4A6D7102
result-dir      = /home1/06758/mschaara/io-500/results/2020.07.12-23.55.34-app
; START 2020-07-12 23:55:43


[ior-easy-write]
t_start         = 2020-07-12 23:55:43
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/stonewall -t 256k -b 20000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
throughput-stonewall = 111759.62
score           = 79.965690
t_delta         = 357.0246
t_end           = 2020-07-13 00:01:40

[mdtest-easy-write]
t_start         = 2020-07-13 00:01:40
exe             = ./mdtest -n 100000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy-stonewall -C -Y -W 300 -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
rate-stonewall  = 11492.710011
score           = 11268.767040
t_delta         = 320.2177
t_end           = 2020-07-13 00:07:00

[timestamp]
t_start         = 2020-07-13 00:07:00
t_delta         = 9.5320
t_end           = 2020-07-13 00:07:10

[ior-hard-write]
t_start         = 2020-07-13 00:07:10
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse
throughput-stonewall = 75603.22
score           = 76.497849
t_delta         = 472.5467
t_end           = 2020-07-13 00:15:02

[mdtest-hard-write]
t_start         = 2020-07-13 00:15:03
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall -C -Y -W 300 -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
rate-stonewall  = 5126.580477
score           = 5030.240860
t_delta         = 327.1818
t_end           = 2020-07-13 00:20:30

[find]
t_start         = 2020-07-13 00:20:30
exe             = mpirun -np 1440 --bind-to socket --hostfile /home1/06758/mschaara/io-500/config/cli_hosts_60 /home1/06758/mschaara/io-500/install/mfu/bin/dfind --pool 695abaf8-862d-44f5-9ae2-296c7a8ea791 --cont f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --svcl 1 --prefix /tmp/dfuse --stonewall 300 /tmp/dfuse/datafiles/2020.07.12-23.55.34-app -newer /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/timestampfile -size 3901 -name "*01*"
last-output     = "MATCHED 243486/2196754994"
total-files     = 2196754994
found           = 243486
score           = 6816.415527
t_delta         = 331.7887
t_end           = 2020-07-13 00:26:02

[ior-easy-read]
t_start         = 2020-07-13 00:26:02
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-easy/stonewall -t 256k -b 20000m -r -R -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse
score           = 74.669062
t_delta         = 388.6528
t_end           = 2020-07-13 00:32:30

[mdtest-easy-stat]
t_start         = 2020-07-13 00:32:30
exe             = ./mdtest -n 100000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy-stonewall -T -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 11157.564332
t_delta         = 301.7934
t_end           = 2020-07-13 00:37:32

[ior-hard-read]
t_start         = 2020-07-13 00:37:32
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.12-23.55.34-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse
score           = 70.759630
t_delta         = 517.9320
t_end           = 2020-07-13 00:46:10

[mdtest-hard-stat]
t_start         = 2020-07-13 00:46:10
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall -T -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 9015.771793
t_delta         = 187.8888
t_end           = 2020-07-13 00:49:18

[mdtest-easy-delete]
t_start         = 2020-07-13 00:49:18
exe             = ./mdtest -n 100000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-easy-stonewall -r -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 6438.142549
t_delta         = 526.7058
t_end           = 2020-07-13 00:58:05

[mdtest-hard-read]
t_start         = 2020-07-13 00:58:05
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall -E -X -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 7372.825241
t_delta         = 227.1045
t_end           = 2020-07-13 01:01:52

[mdtest-hard-delete]
t_start         = 2020-07-13 01:01:52
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.12-23.55.34-app/mdtest-hard-stonewall -r -a DFS --dfs.pool=695abaf8-862d-44f5-9ae2-296c7a8ea791 --dfs.cont=f01034e0-f006-4c7f-9440-8b6fbb5d0a40 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse
score           = 5908.396658
t_delta         = 3907.6244
t_end           = 2020-07-13 02:07:00

[SCORE]
MD              = 7580.227064
BW              = 75.399730
SCORE           = 756.007324 
hash            = 4969E982
; END 2020-07-13 02:07:09
result_summary
IO500 version io500-isc20_v4-22
[RESULT]       ior-easy-write       79.965690 GiB/s  : time 357.025 seconds
[RESULT]    mdtest-easy-write    11268.767040 kIOPS : time 320.218 seconds
[RESULT]       ior-hard-write       76.497849 GiB/s  : time 472.547 seconds
[RESULT]    mdtest-hard-write     5030.240860 kIOPS : time 327.182 seconds
[RESULT]                 find     6816.415527 kIOPS : time 331.789 seconds
[RESULT]        ior-easy-read       74.669062 GiB/s  : time 388.653 seconds
[RESULT]     mdtest-easy-stat    11157.564332 kIOPS : time 301.793 seconds
[RESULT]        ior-hard-read       70.759630 GiB/s  : time 517.932 seconds
[RESULT]     mdtest-hard-stat     9015.771793 kIOPS : time 187.889 seconds
[RESULT]   mdtest-easy-delete     6438.142549 kIOPS : time 526.706 seconds
[RESULT]     mdtest-hard-read     7372.825241 kIOPS : time 227.104 seconds
[RESULT]   mdtest-hard-delete     5908.396658 kIOPS : time 3907.624 seconds
[SCORE] Bandwidth 75.399730 GB/s : IOPS 7580.227064 kiops : TOTAL 756.007324