Presque

Institution Argonne National Laboratory
Client Procs Per Node
Client Operating System RHEL
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.4.3.el7.x86_64

DATA SERVER

Storage Type NVMe-SSD
Volatile Memory 384
Storage Interface NVMe
Network 100gE
Software Version 1.1
OS Version 7.7

INFORMATION

Client Nodes 10
Client Total Procs 380
Metadata Nodes 8
Metadata Storage Devices 12
Data Nodes 8
Data Storage Devices 12

METADATA

Easy Write 3,088.11 kIOP/s
Easy Stat 3,087.05 kIOP/s
Easy Delete 1,786.92 kIOP/s
Hard Write 756.78 kIOP/s
Hard Read 1,476.13 kIOP/s
Hard Stat 2,635.54 kIOP/s
Hard Delete 1,386.67 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Jul 10 12:02:56 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/stonewall -t 256k -b 256g -r -R -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Fri Jul 10 12:02:59 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 380
clients per node    : 38
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 256 GiB
aggregate filesize  : 95 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 104453604638720.
WARNING: Stat() of aggregate file size      = 104295366131712.
WARNING: Using actual aggregate bytes moved = 44322971975680.
read      127505     513907     0.000314    268435456  256.00     2.51       329.01     0.001800   331.51     0   
Max Read:  127505.21 MiB/sec (133698.90 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       127505.21  127505.21  127505.21       0.00  510020.82  510020.82  510020.82       0.00  331.51337         NA            NA     0    380  38    1   0     1        1         0    0      1 274877906944   262144 42269680.0 DFS      0
Finished            : Fri Jul 10 12:08:32 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Jul 10 11:36:50 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/stonewall -t 256k -b 256g -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Fri Jul 10 11:36:53 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 380
clients per node    : 38
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 256 GiB
aggregate filesize  : 95 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 437765 max: 444944 -- min data: 106.9 GiB mean data: 107.8 GiB time: 300.0s
WARNING: Expected aggregate file size       = 104453604638720.
WARNING: Stat() of aggregate file size      = 104295366131712.
WARNING: Using actual aggregate bytes moved = 44322971975680.
WARNING: maybe caused by deadlineForStonewalling
write     137268     557918     0.000287    268435456  256.00     4.88       303.05     0.000613   307.93     0   
Max Write: 137268.27 MiB/sec (143936.21 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      137268.27  137268.27  137268.27       0.00  549073.07  549073.07  549073.07       0.00  307.93482     300.00     139802.97     0    380  38    1   0     1        1         0    0      1 274877906944   262144 42269680.0 DFS      0
Finished            : Fri Jul 10 11:42:02 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Jul 10 12:13:41 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -r -R -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Fri Jul 10 12:13:44 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1600000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 380
clients per node    : 38
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 25.99 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1237382 max: 1237382 -- min data: 54.2 GiB mean data: 54.2 GiB time: 356.4s
WARNING: Expected aggregate file size       = 28580864000000.
WARNING: Stat() of aggregate file size      = 22103404161280.
WARNING: Using actual aggregate bytes moved = 22103404161280.
read      58749      1319151    356.44      45.91      45.91      2.36       356.45     0.001820   358.80     0   
Max Read:  58749.14 MiB/sec (61602.94 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        58749.14   58749.14   58749.14       0.00 1310477.70 1310477.70 1310477.70       0.00  358.80439     356.41      59143.18     0    380  38    1   0     1        1         0    0 1600000    47008    47008 21079448.0 DFS      0
Finished            : Fri Jul 10 12:19:44 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Fri Jul 10 11:47:14 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -w -D 300 -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Fri Jul 10 11:47:17 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1600000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 10
tasks               : 380
clients per node    : 38
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 25.99 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1203966 max: 1237382 -- min data: 52.7 GiB mean data: 53.8 GiB time: 300.0s
WARNING: Expected aggregate file size       = 28580864000000.
WARNING: Stat() of aggregate file size      = 22103404161280.
WARNING: Using actual aggregate bytes moved = 22103404161280.
WARNING: maybe caused by deadlineForStonewalling
write     68190      1544776    300.22      45.91      45.91      4.74       304.38     0.001219   309.13     0   
Max Write: 68190.13 MiB/sec (71502.54 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write       68190.13   68190.13   68190.13       0.00 1521071.71 1521071.71 1521071.71       0.00  309.12754     300.01      69747.76     0    380  38    1   0     1        1         0    0 1600000    47008    47008 21079448.0 DFS      0
Finished            : Fri Jul 10 11:52:28 2020
mdtest-easy-delete
-- started at 07/10/2020 12:21:19 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '4000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy-stonewall' '-r' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 1520000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    1786919.165    1786917.061    1786917.545          0.585
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.067          0.067          0.067          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        527.331        527.330        527.330          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         14.848         14.848         14.848          0.000
-- finished at 07/10/2020 12:30:22 --

mdtest-easy-stat
-- started at 07/10/2020 12:08:35 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '4000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy-stonewall' '-T' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 1520000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    3087051.563    3087045.316    3087049.859          1.590
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        305.242        305.241        305.242          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/10/2020 12:13:40 --

mdtest-easy-write
-- started at 07/10/2020 11:42:05 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '4000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 1520000000 files
Continue stonewall hit min: 2388795 max: 2479726 avg: 2462141.1 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    3088113.873    3088108.410    3088112.291          1.429
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    3144526.472             NA
   Tree creation             :          0.407          0.407          0.407          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        305.137        305.136        305.137          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        297.537             NA
   Tree creation             :          2.455          2.455          2.455          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/10/2020 11:47:13 --

mdtest-hard-delete
-- started at 07/10/2020 12:33:08 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall' '-r' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 570000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    1386670.469    1386667.885    1386668.776          0.762
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.002          0.002          0.002          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        168.766        168.765        168.766          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :        591.758        591.758        591.758          0.000
-- finished at 07/10/2020 12:45:49 --

mdtest-hard-read
-- started at 07/10/2020 12:30:26 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 570000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :    1476131.078    1476128.510    1476129.376          0.672
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        158.538        158.537        158.537          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/10/2020 12:33:04 --

mdtest-hard-stat
-- started at 07/10/2020 12:19:47 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall' '-T' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 570000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    2635539.438    2635535.992    2635537.788          0.969
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         88.795         88.795         88.795          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/10/2020 12:21:16 --

mdtest-hard-write
-- started at 07/10/2020 11:52:31 --

mdtest-3.3.0+dev was launched with 380 total task(s) on 10 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d' '--dfs.cont=b0599c58-2822-45b2-9e09-23a335940082' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 10000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000100000000010000000001000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
380 tasks, 570000000 files
Continue stonewall hit min: 586573 max: 615847 avg: 598328.6 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     756775.745     756773.917     756775.296          0.480
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     757905.470             NA
   Tree creation             :        109.361        109.361        109.361          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        309.236        309.235        309.236          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.991             NA
   Tree creation             :          0.009          0.009          0.009          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/10/2020 11:57:40 --

result
version         = io500-isc20_v4-22
config-hash     = BFF02AC5
result-dir      = ./results/2020.07.10-10.28.13-app
; START 2020-07-10 11:36:50


[ior-easy-write]
t_start         = 2020-07-10 11:36:50
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/stonewall -t 256k -b 256g -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
throughput-stonewall = 50543.60
score           = 134.051044
t_delta         = 312.5268
t_end           = 2020-07-10 11:42:02

[mdtest-easy-write]
t_start         = 2020-07-10 11:42:03
exe             = ./mdtest -n 4000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy-stonewall -C -Y -W 300 -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
rate-stonewall  = 3144.526472
score           = 3088.112336
t_delta         = 311.4472
t_end           = 2020-07-10 11:47:14

[timestamp]
t_start         = 2020-07-10 11:47:14
t_delta         = 0.2608
t_end           = 2020-07-10 11:47:14

[ior-hard-write]
t_start         = 2020-07-10 11:47:14
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -w -D 300 -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
throughput-stonewall = 25119.28
score           = 66.591929
t_delta         = 313.3325
t_end           = 2020-07-10 11:52:28

[mdtest-hard-write]
t_start         = 2020-07-10 11:52:28
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall -C -Y -W 300 -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
rate-stonewall  = 757.905470
score           = 756.775345
t_delta         = 313.0488
t_end           = 2020-07-10 11:57:41

[find]
t_start         = 2020-07-10 11:57:41
exe             = mpirun --bind-to socket -np 380 -f /home/kalfizah/io500-new/hfile /home/kalfizah/mfu/install/bin/dfind --pool a04776a5-bd54-4bf9-ae46-c989b9407c8d --cont b0599c58-2822-45b2-9e09-23a335940082 --svcl 1 --prefix /tmp/dfuse/ --stonewall 300 /tmp/dfuse/datafiles/2020.07.10-10.28.13-app -newer /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/timestampfile -size 3901 -name "*01*"
last-output     = "MATCHED 1822809/446073739"
total-files     = 446073739
found           = 1822809
score           = 1415.441473
t_delta         = 315.3989
t_end           = 2020-07-10 12:02:56

[ior-easy-read]
t_start         = 2020-07-10 12:02:56
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-easy/stonewall -t 256k -b 256g -r -R -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
score           = 124.516802
t_delta         = 335.4039
t_end           = 2020-07-10 12:08:32

[mdtest-easy-stat]
t_start         = 2020-07-10 12:08:32
exe             = ./mdtest -n 4000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy-stonewall -T -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 3087.050375
t_delta         = 309.1387
t_end           = 2020-07-10 12:13:41

[ior-hard-read]
t_start         = 2020-07-10 12:13:41
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.10-10.28.13-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -r -R -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
score           = 57.372205
t_delta         = 362.7710
t_end           = 2020-07-10 12:19:44

[mdtest-hard-stat]
t_start         = 2020-07-10 12:19:44
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall -T -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 2635.539063
t_delta         = 92.5986
t_end           = 2020-07-10 12:21:16

[mdtest-easy-delete]
t_start         = 2020-07-10 12:21:17
exe             = ./mdtest -n 4000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-easy-stonewall -r -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 1786.917367
t_delta         = 546.0172
t_end           = 2020-07-10 12:30:23

[mdtest-hard-read]
t_start         = 2020-07-10 12:30:23
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall -E -X -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 1476.129455
t_delta         = 162.4424
t_end           = 2020-07-10 12:33:05

[mdtest-hard-delete]
t_start         = 2020-07-10 12:33:05
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.10-10.28.13-app/mdtest-hard-stonewall -r -a DFS --dfs.pool=a04776a5-bd54-4bf9-ae46-c989b9407c8d --dfs.cont=b0599c58-2822-45b2-9e09-23a335940082 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 1386.668114
t_delta         = 764.3861
t_end           = 2020-07-10 12:45:50

[SCORE]
MD              = 1774.785309
BW              = 89.362503
SCORE           = 398.245222 
hash            = B892004
; END 2020-07-10 12:45:50
result_summary
IO500 version io500-isc20_v4-22
[RESULT]       ior-easy-write      134.051044 GiB/s  : time 312.527 seconds
[RESULT]    mdtest-easy-write     3088.112336 kIOPS : time 311.447 seconds
[RESULT]       ior-hard-write       66.591929 GiB/s  : time 313.333 seconds
[RESULT]    mdtest-hard-write      756.775345 kIOPS : time 313.049 seconds
[RESULT]                 find     1415.441473 kIOPS : time 315.399 seconds
[RESULT]        ior-easy-read      124.516802 GiB/s  : time 335.404 seconds
[RESULT]     mdtest-easy-stat     3087.050375 kIOPS : time 309.139 seconds
[RESULT]        ior-hard-read       57.372205 GiB/s  : time 362.771 seconds
[RESULT]     mdtest-hard-stat     2635.539063 kIOPS : time 92.599 seconds
[RESULT]   mdtest-easy-delete     1786.917367 kIOPS : time 546.017 seconds
[RESULT]     mdtest-hard-read     1476.129455 kIOPS : time 162.442 seconds
[RESULT]   mdtest-hard-delete     1386.668114 kIOPS : time 764.386 seconds
[SCORE] Bandwidth 89.362503 GB/s : IOPS 1774.785309 kiops : TOTAL 398.245222