Wolf

Institution Intel
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.12.1.el7.x86_64

DATA SERVER

Storage Type Optane Persistent Memory
Volatile Memory 512GB
Storage Interface DDR-T
Network Omnipath
Software Version 1.1
OS Version 7.7

INFORMATION

Client Nodes 52
Client Total Procs 1,664
Metadata Nodes 0
Metadata Storage Devices 0
Data Nodes 30
Data Storage Devices 60

METADATA

Easy Write 11,977.14 kIOP/s
Easy Stat 8,189.87 kIOP/s
Easy Delete 5,496.88 kIOP/s
Hard Write 5,470.99 kIOP/s
Hard Read 7,472.00 kIOP/s
Hard Stat 7,910.95 kIOP/s
Hard Delete 5,396.35 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Jun 28 22:15:35 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/stonewall -t 128k -b 50000m -r -R -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-134.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Sun Jun 28 22:15:37 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 52
tasks               : 1664
clients per node    : 32
repetitions         : 1
xfersize            : 131072 bytes
blocksize           : 48.83 GiB
aggregate filesize  : 79.35 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
read      589973     4725525    0.000352    51200000   128.00     0.167629   140.85     0.001752   141.02     0   
Max Read:  589973.31 MiB/sec (618631.85 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       589973.31  589973.31  589973.31       0.00 4719786.49 4719786.49 4719786.49       0.00  141.02333         NA            NA     0   1664  32    1   0     1        1         0    0      1 52428800000   131072 83200000.0 DFS      0
Finished            : Sun Jun 28 22:18:32 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Jun 28 21:44:53 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/stonewall -t 128k -b 50000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-134.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Sun Jun 28 21:44:55 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 52
tasks               : 1664
clients per node    : 32
repetitions         : 1
xfersize            : 131072 bytes
blocksize           : 48.83 GiB
aggregate filesize  : 79.35 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 337178 max: 400000 -- min data: 41.2 GiB mean data: 44.4 GiB time: 300.4s
write     250175     2002073    0.000751    51200000   128.00     0.111442   332.46     0.000414   332.57     0   
Max Write: 250174.81 MiB/sec (262327.30 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      250174.81  250174.81  250174.81       0.00 2001398.48 2001398.48 2001398.48       0.00  332.56746     300.42     252088.34     0   1664  32    1   0     1        1         0    0      1 52428800000   131072 83200000.0 DFS      0
Finished            : Sun Jun 28 21:51:02 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Jun 28 22:27:05 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-134.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Sun Jun 28 22:27:07 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 52
tasks               : 1664
clients per node    : 32
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 711.42 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1131798 max: 1131798 -- min data: 49.5 GiB mean data: 49.5 GiB time: 168.1s
WARNING: Expected aggregate file size       = 782213120000000.
WARNING: Stat() of aggregate file size      = 88530724478976.
WARNING: Using actual aggregate bytes moved = 88530724478976.
read      500988     11186430   168.36      45.91      45.91      0.168421   168.36     0.000602   168.53     0   
Max Read:  500988.46 MiB/sec (525324.48 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       500988.46  500988.46  500988.46       0.00 11175214.36 11175214.36 11175214.36       0.00  168.52579     168.07     502346.43     0   1664  32    1   0     1        1         0    0 10000000    47008    47008 84429480.0 DFS      0
Finished            : Sun Jun 28 22:30:29 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sun Jun 28 21:57:04 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
Machine             : Linux wolf-134.wolf.hpdd.intel.com
TestID              : 0
StartTime           : Sun Jun 28 21:57:06 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 10000000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 52
tasks               : 1664
clients per node    : 32
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 711.42 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1083044 max: 1131798 -- min data: 47.4 GiB mean data: 49.1 GiB time: 300.2s
WARNING: Expected aggregate file size       = 782213120000000.
WARNING: Stat() of aggregate file size      = 88530724478976.
WARNING: Using actual aggregate bytes moved = 88530724478976.
WARNING: maybe caused by deadlineForStonewalling
write     274003     6116257    301.27      45.91      45.91      0.214048   307.92     0.000537   308.13     0   
Max Write: 274002.82 MiB/sec (287312.78 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      274002.82  274002.82  274002.82       0.00 6111997.46 6111997.46 6111997.46       0.00  308.13362     300.24     278491.27     0   1664  32    1   0     1        1         0    0 10000000    47008    47008 84429480.0 DFS      0
Finished            : Sun Jun 28 22:02:48 2020
mdtest-easy-delete
-- started at 06/28/2020 22:34:51 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy-stonewall' '-r' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    5496882.273    5496866.040    5496874.220          3.492
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.062          0.062          0.062          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        709.380        709.378        709.379          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         16.150         16.150         16.150          0.000
-- finished at 06/28/2020 22:46:57 --

mdtest-easy-stat
-- started at 06/28/2020 22:18:35 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy-stonewall' '-T' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    8189894.215    8189851.711    8189873.173          9.775
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        476.122        476.120        476.121          0.001
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/28/2020 22:26:31 --

mdtest-easy-write
-- started at 06/28/2020 21:51:04 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files
Continue stonewall hit min: 2087119 max: 2343371 avg: 2249223.5 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :   11977158.221   11977134.167   11977146.173          6.541
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA   12471503.973             NA
   Tree creation             :          3.991          3.991          3.991          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        325.568        325.567        325.567          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.101             NA
   Tree creation             :          0.251          0.251          0.251          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/28/2020 21:56:30 --

mdtest-hard-delete
-- started at 06/28/2020 22:52:06 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall' '-r' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    5396364.562    5396326.588    5396345.380          8.892
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        327.325        327.322        327.324          0.001
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :       4190.711       4190.711       4190.711          0.000
-- finished at 06/29/2020 00:07:24 --

mdtest-hard-read
-- started at 06/28/2020 22:47:33 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :    7472018.897    7471959.987    7471989.158         13.769
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        236.397        236.395        236.396          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/28/2020 22:51:30 --

mdtest-hard-stat
-- started at 06/28/2020 22:30:32 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall' '-T' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    7911008.700    7910933.389    7910969.069         17.562
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        223.280        223.278        223.279          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/28/2020 22:34:15 --

mdtest-hard-write
-- started at 06/28/2020 22:02:50 --

mdtest-3.3.0+dev was launched with 1664 total task(s) on 52 node(s)
Command line used: ./mdtest '-n' '10000000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard' '-x' '/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459' '--dfs.cont=df387677-10df-43c0-8ace-2c11885f8992' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/mnt/dfuse/'
Nodemap: 10000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
1664 tasks, 16640000000 files
Continue stonewall hit min: 961392 max: 1061509 avg: 1012253.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    5471013.067    5470993.702    5471006.539          7.048
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    5607000.575             NA
   Tree creation             :        519.174        519.174        519.174          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        322.857        322.856        322.857          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.408             NA
   Tree creation             :          0.002          0.002          0.002          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 06/28/2020 22:08:13 --

result
version         = io500-isc20-1-g643b0ae16ce8-22
config-hash     = F8597C07
result-dir      = /home/daos/io-500/results/2020.06.28-19.21.15-app
; START 2020-06-28 21:44:53


[ior-easy-write]
t_start         = 2020-06-28 21:44:53
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/stonewall -t 128k -b 50000m -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
throughput-stonewall = 370046.86
score           = 244.311338
t_delta         = 368.6671
t_end           = 2020-06-28 21:51:02

[mdtest-easy-write]
t_start         = 2020-06-28 21:51:02
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy-stonewall -C -Y -W 300 -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
rate-stonewall  = 12471.503973
score           = 11977.136345
t_delta         = 361.9100
t_end           = 2020-06-28 21:57:04

[timestamp]
t_start         = 2020-06-28 21:57:04
t_delta         = 0.0174
t_end           = 2020-06-28 21:57:04

[ior-hard-write]
t_start         = 2020-06-28 21:57:04
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -w -D 300 -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
throughput-stonewall = 440960.42
score           = 267.580875
t_delta         = 344.1219
t_end           = 2020-06-28 22:02:48

[mdtest-hard-write]
t_start         = 2020-06-28 22:02:48
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall -C -Y -W 300 -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
rate-stonewall  = 5607.000575
score           = 5470.994206
t_delta         = 358.5828
t_end           = 2020-06-28 22:08:46

[find]
t_start         = 2020-06-28 22:08:46
exe             = mpirun -np 1664 -iface ib0 --bind-to socket --hostfile /home/daos/io-500/config/cli_hosts_full /home/daos/io-500/install/mfu/bin/dfind --pool fe1349ec-d68a-486c-b13e-440c0a17f459 --cont df387677-10df-43c0-8ace-2c11885f8992 --svcl 1 --prefix /mnt/dfuse/ --stonewall 300 /mnt/dfuse/datafiles/2020.06.28-19.21.15-app -newer /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/timestampfile -size 3901 -name "*01*"
nproc           = 1
last-output     = "MATCHED 121034/2709897522"
total-files     = 2709897522
found           = 121034
score           = 6637.833981
t_delta         = 408.2634
t_end           = 2020-06-28 22:15:35

[ior-easy-read]
t_start         = 2020-06-28 22:15:35
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-easy/stonewall -t 128k -b 50000m -r -R -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.prefix=/mnt/dfuse/
score           = 576.145812
t_delta         = 177.4884
t_end           = 2020-06-28 22:18:32

[mdtest-easy-stat]
t_start         = 2020-06-28 22:18:32
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy-stonewall -T -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 8189.868412
t_delta         = 512.3564
t_end           = 2020-06-28 22:27:04

[ior-hard-read]
t_start         = 2020-06-28 22:27:04
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/file -O stoneWallingStatusFile=/mnt/dfuse/datafiles/2020.06.28-19.21.15-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 10000000 -r -R -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/mnt/dfuse/
score           = 489.246544
t_delta         = 204.9824
t_end           = 2020-06-28 22:30:29

[mdtest-hard-stat]
t_start         = 2020-06-28 22:30:29
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall -T -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 7910.949853
t_delta         = 259.3190
t_end           = 2020-06-28 22:34:49

[mdtest-easy-delete]
t_start         = 2020-06-28 22:34:49
exe             = ./mdtest -n 10000000 -u -L -F -P -N 1 -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-easy-stonewall -r -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 5496.875930
t_delta         = 761.7271
t_end           = 2020-06-28 22:47:30

[mdtest-hard-read]
t_start         = 2020-06-28 22:47:30
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall -E -X -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 7471.995663
t_delta         = 272.5126
t_end           = 2020-06-28 22:52:03

[mdtest-hard-delete]
t_start         = 2020-06-28 22:52:03
exe             = ./mdtest -n 10000000 -t -w 3901 -e 3901 -P -N 1 -F -d /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard -x /mnt/dfuse/datafiles/2020.06.28-19.21.15-app/mdtest-hard-stonewall -r -a DFS --dfs.pool=fe1349ec-d68a-486c-b13e-440c0a17f459 --dfs.cont=df387677-10df-43c0-8ace-2c11885f8992 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/mnt/dfuse/
score           = 5396.353407
t_delta         = 4554.6348
t_end           = 2020-06-29 00:07:58

[SCORE]
MD              = 7070.506322
BW              = 368.438255
SCORE           = 1614.015183 
hash            = 8BA5F05B
; END 2020-06-29 00:07:58
result_summary
IO500 version io500-isc20-1-g643b0ae16ce8-22
[RESULT]       ior-easy-write      244.311338 GiB/s  : time 368.667 seconds
[RESULT]    mdtest-easy-write    11977.136345 kIOPS : time 361.910 seconds
[RESULT]       ior-hard-write      267.580875 GiB/s  : time 344.122 seconds
[RESULT]    mdtest-hard-write     5470.994206 kIOPS : time 358.583 seconds
[RESULT]                 find     6637.833981 kIOPS : time 408.263 seconds
[RESULT]        ior-easy-read      576.145812 GiB/s  : time 177.488 seconds
[RESULT]     mdtest-easy-stat     8189.868412 kIOPS : time 512.356 seconds
[RESULT]        ior-hard-read      489.246544 GiB/s  : time 204.982 seconds
[RESULT]     mdtest-hard-stat     7910.949853 kIOPS : time 259.319 seconds
[RESULT]   mdtest-easy-delete     5496.875930 kIOPS : time 761.727 seconds
[RESULT]     mdtest-hard-read     7471.995663 kIOPS : time 272.513 seconds
[RESULT]   mdtest-hard-delete     5396.353407 kIOPS : time 4554.635 seconds
[SCORE] Bandwidth 368.438255 GB/s : IOPS 7070.506322 kiops : TOTAL 1614.015183