Wolf

Institution Intel
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.12.1.el7.x86_64

DATA SERVER

Storage Type Optane Persistent Memory
Volatile Memory 512GB
Storage Interface DDR-T
Network Omnipath
Software Version 1.1
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 420
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 30
Data Storage Devices 60

METADATA

Easy Write 4,707.23 kIOP/s
Easy Stat 4,392.86 kIOP/s
Easy Delete 2,745.02 kIOP/s
Hard Write 2,326.17 kIOP/s
Hard Read 3,164.63 kIOP/s
Hard Stat 3,962.69 kIOP/s
Hard Delete 2,801.64 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jun 29 06:49:05 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/stonewall -t 128k -b 200000m -r -R -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-79.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Mon Jun 29 06:49:08 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 420
clients per node    : 42
repetitions         : 1
xfersize            : 131072 bytes
blocksize           : 195.31 GiB
aggregate filesize  : 80.11 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 88080384000000.
WARNING: Stat() of aggregate file size      = 88012048826368.
WARNING: Using actual aggregate bytes moved = 59379611074560.
read      209223     1673910    0.000169    204800000  128.00     0.020050   270.64     0.000413   270.66     0   
Max Read:  209222.64 MiB/sec (219385.84 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       209222.64  209222.64  209222.64       0.00 1673781.13 1673781.13 1673781.13       0.00  270.66291         NA            NA     0    420  42    1   0     1        1         0    0      1 209715200000   131072 56628808.0 DFS      0
Finished            : Mon Jun 29 06:53:47 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jun 29 06:21:25 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/stonewall -t 128k -b 200000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-79.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Mon Jun 29 06:21:28 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 420
clients per node    : 42
repetitions         : 1
xfersize            : 131072 bytes
blocksize           : 195.31 GiB
aggregate filesize  : 80.11 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1022379 max: 1078644 -- min data: 124.8 GiB mean data: 127.9 GiB time: 300.2s
WARNING: Expected aggregate file size       = 88080384000000.
WARNING: Stat() of aggregate file size      = 88012048826368.
WARNING: Using actual aggregate bytes moved = 59379611074560.
WARNING: maybe caused by deadlineForStonewalling
write     180572     1444740    0.000196    204800000  128.00     0.035780   313.57     0.000475   313.61     0   
Max Write: 180571.54 MiB/sec (189342.98 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      180571.54  180571.54  180571.54       0.00 1444572.32 1444572.32 1444572.32       0.00  313.60872     300.21     183247.69     0    420  42    1   0     1        1         0    0      1 209715200000   131072 56628808.0 DFS      0
Finished            : Mon Jun 29 06:26:50 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jun 29 06:59:50 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-79.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Mon Jun 29 06:59:53 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 420
clients per node    : 42
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 179.56 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 2019906 max: 2019906 -- min data: 88.4 GiB mean data: 88.4 GiB time: 225.6s
WARNING: Expected aggregate file size       = 197433600000000.
WARNING: Stat() of aggregate file size      = 39879731324160.
WARNING: Using actual aggregate bytes moved = 39879731324160.
read      168257     3753545    226.01      45.91      45.91      0.020035   226.02     0.000385   226.04     0   
Max Read:  168257.40 MiB/sec (176430.67 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       168257.40  168257.40  168257.40       0.00 3753205.13 3753205.13 3753205.13       0.00  226.03628     225.57     168603.39     0    420  42    1   0     1        1         0    0 10000000    47008    47008 38032276.0 DFS      0
Finished            : Mon Jun 29 07:03:48 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Mon Jun 29 06:32:29 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-79.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Mon Jun 29 06:32:32 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 420
clients per node    : 42
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 179.56 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1948547 max: 2019906 -- min data: 85.3 GiB mean data: 87.1 GiB time: 300.2s
WARNING: Expected aggregate file size       = 197433600000000.
WARNING: Stat() of aggregate file size      = 39879731324160.
WARNING: Using actual aggregate bytes moved = 39879731324160.
WARNING: maybe caused by deadlineForStonewalling
write     122674     2736669    302.82      45.91      45.91      0.030024   310.00     0.000320   310.03     0   
Max Write: 122673.68 MiB/sec (128632.68 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      122673.68  122673.68  122673.68       0.00 2736399.75 2736399.75 2736399.75       0.00  310.02799     300.21     124728.22     0    420  42    1   0     1        1         0    0 10000000    47008    47008 38032276.0 DFS      0
Finished            : Mon Jun 29 06:37:50 2020
mdtest-easy-delete
-- started at 06/29/2020 07:07:14 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy-stonewall' '-r' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    2745023.564    2745020.115    2745021.607          1.038
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.093          0.093          0.093          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        561.429        561.428        561.428          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         10.734         10.734         10.734          0.000
-- finished at 06/29/2020 07:16:46 --

mdtest-easy-stat
-- started at 06/29/2020 06:53:51 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy-stonewall' '-T' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    4392856.960    4392845.081    4392850.638          3.267
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        350.828        350.827        350.828          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/29/2020 06:59:41 --

mdtest-easy-write
-- started at 06/29/2020 06:26:53 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files
Continue stonewall hit min: 3334056 max: 3669364 avg: 3594501.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    4707230.906    4707229.364    4707230.298          0.410
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    5026145.756             NA
   Tree creation             :         24.417         24.417         24.417          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        327.397        327.397        327.397          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.367             NA
   Tree creation             :          0.041          0.041          0.041          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/29/2020 06:32:20 --

mdtest-hard-delete
-- started at 06/29/2020 07:21:09 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall' '-r' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    2801645.112    2801637.222    2801640.426          2.403
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.001          0.001          0.001          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        270.666        270.665        270.665          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :       1793.044       1793.044       1793.044          0.000
-- finished at 06/29/2020 07:55:33 --

mdtest-hard-read
-- started at 06/29/2020 07:16:58 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :    3164628.165    3164619.339    3164622.984          2.729
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        239.620        239.620        239.620          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/29/2020 07:20:58 --

mdtest-hard-stat
-- started at 06/29/2020 07:03:51 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall' '-T' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    3962688.852    3962674.008    3962680.591          4.544
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        191.362        191.362        191.362          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/29/2020 07:07:02 --

mdtest-hard-write
-- started at 06/29/2020 06:37:53 --

mdtest-3.3.0+dev was launched with 420 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442' '--dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
420 tasks, 4200000000 files
Continue stonewall hit min: 1653715 max: 1805493 avg: 1764976.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    2326173.175    2326172.240    2326172.832          0.242
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    2467629.985             NA
   Tree creation             :        507.321        507.321        507.321          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        325.989        325.989        325.989          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.406             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/29/2020 06:43:19 --

result
version         = io500-isc20-1-g643b0ae16ce8-22
config-hash     = D9FF7FE8
result-dir      = /home/daos/io-500/results/2020.06.29-04.46.44-app
; START 2020-06-29 06:21:25


[ior-easy-write]
t_start         = 2020-06-29 06:21:25
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/stonewall -t 128k -b 200000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
throughput-stonewall = 71948.70
score           = 176.339395
t_delta         = 325.2235
t_end           = 2020-06-29 06:26:50

[mdtest-easy-write]
t_start         = 2020-06-29 06:26:50
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy-stonewall -C -Y -W 300 -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
rate-stonewall  = 5026.145756
score           = 4707.230358
t_delta         = 338.8832
t_end           = 2020-06-29 06:32:29

[timestamp]
t_start         = 2020-06-29 06:32:29
t_delta         = 0.0133
t_end           = 2020-06-29 06:32:29

[ior-hard-write]
t_start         = 2020-06-29 06:32:29
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
throughput-stonewall = 49538.12
score           = 119.798518
t_delta         = 321.5567
t_end           = 2020-06-29 06:37:50

[mdtest-hard-write]
t_start         = 2020-06-29 06:37:50
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall -C -Y -W 300 -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
rate-stonewall  = 2467.629985
score           = 2326.172983
t_delta         = 337.4270
t_end           = 2020-06-29 06:43:28

[find]
t_start         = 2020-06-29 06:43:28
exe             = mpirun -np 420 -iface ib0 --bind-to socket --hostfile /home/daos/io-500/config/cli_hosts /home/daos/io-500/install/mfu/bin/dfind --pool 9ae1db5f-54fa-4822-81cd-2faaaeacc442 --cont 9f8cf146-1aec-46d8-8789-5a0746355be1 --svcl 1 --prefix /mnt/dfuse/ --stonewall 300 /mnt/dfuse/datafiles/2020.06.29-04.46.44-app -newer /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/timestampfile -size 3901 -name "*01*"
nproc           = 1
last-output     = "MATCHED 254663/1079267961"
total-files     = 1079267961
found           = 254663
score           = 3200.326413
t_delta         = 337.2495
t_end           = 2020-06-29 06:49:05

[ior-easy-read]
t_start         = 2020-06-29 06:49:05
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-easy/stonewall -t 128k -b 200000m -r -R -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
score           = 204.318986
t_delta         = 282.3626
t_end           = 2020-06-29 06:53:47

[mdtest-easy-stat]
t_start         = 2020-06-29 06:53:47
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy-stonewall -T -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 4392.856960
t_delta         = 362.5309
t_end           = 2020-06-29 06:59:50

[ior-hard-read]
t_start         = 2020-06-29 06:59:50
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.29-04.46.44-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
score           = 164.313863
t_delta         = 237.8284
t_end           = 2020-06-29 07:03:48

[mdtest-hard-stat]
t_start         = 2020-06-29 07:03:48
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall -T -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 3962.688852
t_delta         = 202.9795
t_end           = 2020-06-29 07:07:11

[mdtest-easy-delete]
t_start         = 2020-06-29 07:07:11
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-easy-stonewall -r -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 2745.023547
t_delta         = 584.0620
t_end           = 2020-06-29 07:16:55

[mdtest-hard-read]
t_start         = 2020-06-29 07:16:55
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall -E -X -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 3164.628165
t_delta         = 251.3198
t_end           = 2020-06-29 07:21:06

[mdtest-hard-delete]
t_start         = 2020-06-29 07:21:06
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.29-04.46.44-app/mdtest-hard-stonewall -r -a DFS --dfs.pool=9ae1db5f-54fa-4822-81cd-2faaaeacc442 --dfs.cont=9f8cf146-1aec-46d8-8789-5a0746355be1 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 2801.637532
t_delta         = 2075.6502
t_end           = 2020-06-29 07:55:42

[SCORE]
MD              = 3322.403955
BW              = 163.190897
SCORE           = 736.332861 
hash            = 60DEF1A0
; END 2020-06-29 07:55:42
result_summary
IO500 version io500-isc20-1-g643b0ae16ce8-22
[RESULT]       ior-easy-write      176.339395 GiB/s  : time 325.223 seconds
[RESULT]    mdtest-easy-write     4707.230358 kIOPS : time 338.883 seconds
[RESULT]       ior-hard-write      119.798518 GiB/s  : time 321.557 seconds
[RESULT]    mdtest-hard-write     2326.172983 kIOPS : time 337.427 seconds
[RESULT]                 find     3200.326413 kIOPS : time 337.249 seconds
[RESULT]        ior-easy-read      204.318986 GiB/s  : time 282.363 seconds
[RESULT]     mdtest-easy-stat     4392.856960 kIOPS : time 362.531 seconds
[RESULT]        ior-hard-read      164.313863 GiB/s  : time 237.828 seconds
[RESULT]     mdtest-hard-stat     3962.688852 kIOPS : time 202.979 seconds
[RESULT]   mdtest-easy-delete     2745.023547 kIOPS : time 584.062 seconds
[RESULT]     mdtest-hard-read     3164.628165 kIOPS : time 251.320 seconds
[RESULT]   mdtest-hard-delete     2801.637532 kIOPS : time 2075.650 seconds
[SCORE] Bandwidth 163.190897 GB/s : IOPS 3322.403955 kiops : TOTAL 736.332861