Presque

Institution Argonne National Laboratory
Client Procs Per Node
Client Operating System RHEL
Client Operating System Version 7.7
Client Kernel Version 3.10.0-1062.4.3.el7.x86_64

DATA SERVER

Storage Type NVMe-SSD
Volatile Memory 384
Storage Interface NVMe
Network 100gE
Software Version 1.1
OS Version 7.7

INFORMATION

Client Nodes 16
Client Total Procs 544
Metadata Nodes 10
Metadata Storage Devices 12
Data Nodes 10
Data Storage Devices 12

METADATA

Easy Write 4,240.89 kIOP/s
Easy Stat 4,185.47 kIOP/s
Easy Delete 2,249.35 kIOP/s
Hard Write 813.60 kIOP/s
Hard Read 1,918.11 kIOP/s
Hard Stat 3,430.21 kIOP/s
Hard Delete 2,188.87 kIOP/s

Submitted Files

ior-easy-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 21:39:36 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/stonewall -t 256k -b 256g -r -R -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Thu Jul  9 21:39:39 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 16
tasks               : 544
clients per node    : 34
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 256 GiB
aggregate filesize  : 136 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
WARNING: Expected aggregate file size       = 149533581377536.
WARNING: Stat() of aggregate file size      = 149342991417344.
WARNING: Using actual aggregate bytes moved = 45852643033088.
read      130983     530529     0.000314    268435456  256.00     4.15       329.70     0.001316   333.85     0   
Max Read:  130982.92 MiB/sec (137345.54 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       130982.92  130982.92  130982.92       0.00  523931.67  523931.67  523931.67       0.00  333.84879         NA            NA     0    544  34    1   0     1        1         0    0      1 274877906944   262144 43728488.0 DFS      0
Finished            : Thu Jul  9 21:45:14 2020
ior-easy-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 21:11:28 2020
Command line        : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/stonewall -t 256k -b 256g -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Thu Jul  9 21:11:30 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/ior_file_easy
access              : single-shared-file
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 16
tasks               : 544
clients per node    : 34
repetitions         : 1
xfersize            : 262144 bytes
blocksize           : 256 GiB
aggregate filesize  : 136 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 177261 max: 321533 -- min data: 43.3 GiB mean data: 60.4 GiB time: 300.0s
WARNING: Expected aggregate file size       = 149533581377536.
WARNING: Stat() of aggregate file size      = 149342991417344.
WARNING: Using actual aggregate bytes moved = 45852643033088.
WARNING: maybe caused by deadlineForStonewalling
write     103464     420775     0.000298    268435456  256.00     6.95       415.69     0.000651   422.65     0   
Max Write: 103463.83 MiB/sec (108489.69 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      103463.83  103463.83  103463.83       0.00  413855.31  413855.31  413855.31       0.00  422.64518     300.03     112088.55     0    544  34    1   0     1        1         0    0      1 274877906944   262144 43728488.0 DFS      0
Finished            : Thu Jul  9 21:18:35 2020
ior-hard-read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 21:50:26 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -r -R -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Thu Jul  9 21:50:29 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1600000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 16
tasks               : 544
clients per node    : 34
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 37.21 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1393909 max: 1393909 -- min data: 61.0 GiB mean data: 61.0 GiB time: 410.2s
WARNING: Expected aggregate file size       = 40915763200000.
WARNING: Stat() of aggregate file size      = 35645531603968.
WARNING: Using actual aggregate bytes moved = 35645531603968.
read      81923      1848447    410.23      45.91      45.91      4.72       410.23     0.001403   414.95     0   
Max Read:  81922.94 MiB/sec (85902.42 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read        81922.94   81922.94   81922.94       0.00 1827400.11 1827400.11 1827400.11       0.00  414.95373     410.20      82872.69     0    544  34    1   0     1        1         0    0 1600000    47008    47008 33994228.0 DFS      0
Finished            : Thu Jul  9 21:57:25 2020
ior-hard-write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Thu Jul  9 21:23:48 2020
Command line        : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -w -D 300 -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
Machine             : Linux daos08.ftm.alcf.anl.gov
TestID              : 0
StartTime           : Thu Jul  9 21:23:50 2020

Options: 
api                 : DFS
apiVersion          : DAOS
test filename       : /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/file
access              : single-shared-file
type                : independent
segments            : 1600000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 16
tasks               : 544
clients per node    : 34
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 37.21 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
stonewalling pairs accessed min: 1360440 max: 1393909 -- min data: 59.6 GiB mean data: 60.8 GiB time: 300.0s
WARNING: Expected aggregate file size       = 40915763200000.
WARNING: Stat() of aggregate file size      = 35645531603968.
WARNING: Using actual aggregate bytes moved = 35645531603968.
WARNING: maybe caused by deadlineForStonewalling
write     109479     2495561    300.81      45.91      45.91      6.65       303.85     0.000956   310.51     0   
Max Write: 109479.29 MiB/sec (114797.36 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      109479.29  109479.29  109479.29       0.00 2442081.27 2442081.27 2442081.27       0.00  310.50830     300.01     112981.68     0    544  34    1   0     1        1         0    0 1600000    47008    47008 33994228.0 DFS      0
Finished            : Thu Jul  9 21:29:03 2020
mdtest-easy-delete
-- started at 07/09/2020 21:58:47 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '4000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy-stonewall' '-r' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 2176000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    2249347.484    2249344.808    2249345.748          0.759
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.066          0.066          0.066          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        572.830        572.829        572.829          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :         15.261         15.261         15.261          0.000
-- finished at 07/09/2020 22:08:35 --

mdtest-easy-stat
-- started at 07/09/2020 21:45:17 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '4000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy-stonewall' '-T' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 2176000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    4185476.943    4185471.539    4185474.117          1.567
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        307.849        307.848        307.848          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 21:50:25 --

mdtest-easy-write
-- started at 07/09/2020 21:18:38 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '4000000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 2176000000 files
Continue stonewall hit min: 2272069 max: 2368550 avg: 2344496.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :    4240899.401    4240890.958    4240896.149          2.185
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA    4318818.381             NA
   Tree creation             :          0.214          0.214          0.214          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        303.826        303.825        303.825          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        295.314             NA
   Tree creation             :          4.669          4.669          4.669          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 21:23:46 --

mdtest-hard-delete
-- started at 07/09/2020 22:10:56 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall' '-r' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 816000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :    2188871.012    2188867.069    2188869.069          1.098
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.002          0.002          0.002          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        116.082        116.081        116.081          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :        614.565        614.565        614.565          0.000
-- finished at 07/09/2020 22:23:06 --

mdtest-hard-read
-- started at 07/09/2020 22:08:39 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 816000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :    1918108.293    1918104.917    1918106.230          0.964
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        132.468        132.468        132.468          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 22:10:51 --

mdtest-hard-stat
-- started at 07/09/2020 21:57:28 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall' '-T' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 816000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :    3430211.614    3430203.234    3430207.714          2.460
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :         74.074         74.073         74.073          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 21:58:42 --

mdtest-hard-write
-- started at 07/09/2020 21:29:06 --

mdtest-3.3.0+dev was launched with 544 total task(s) on 16 node(s)
Command line used: ./mdtest '-n' '1500000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard' '-x' '/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'DFS' '--dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb' '--dfs.cont=ce610953-295d-464c-af58-bd04279ded70' '--dfs.svcl=1' '--dfs.oclass=S1' '--dfs.prefix=/tmp/dfuse/'
Nodemap: 1000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000100000000000000010000000000000001000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 1 for each phase.
544 tasks, 816000000 files
Continue stonewall hit min: 436077 max: 467072 avg: 446985.2 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     813604.137     813603.402     813603.686          0.227
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     810555.174             NA
   Tree creation             :        126.266        126.266        126.266          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        312.299        312.298        312.298          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.992             NA
   Tree creation             :          0.008          0.008          0.008          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/09/2020 21:34:18 --

result
version         = io500-isc20_v4-22
config-hash     = C095E7AF
result-dir      = ./results/2020.07.09-19.56.01-app
; START 2020-07-09 21:11:28


[ior-easy-write]
t_start         = 2020-07-09 21:11:28
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/stonewall -t 256k -b 256g -w -D 300 -O stoneWallingWearOut=1 -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
throughput-stonewall = 42271.79
score           = 101.038894
t_delta         = 426.7161
t_end           = 2020-07-09 21:18:35

[mdtest-easy-write]
t_start         = 2020-07-09 21:18:35
exe             = ./mdtest -n 4000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy-stonewall -C -Y -W 300 -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
rate-stonewall  = 4318.818381
score           = 4240.894878
t_delta         = 312.3422
t_end           = 2020-07-09 21:23:47

[timestamp]
t_start         = 2020-07-09 21:23:47
t_delta         = 0.5722
t_end           = 2020-07-09 21:23:48

[ior-hard-write]
t_start         = 2020-07-09 21:23:48
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -w -D 300 -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
throughput-stonewall = 57991.23
score           = 106.913370
t_delta         = 314.9644
t_end           = 2020-07-09 21:29:03

[mdtest-hard-write]
t_start         = 2020-07-09 21:29:03
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall -C -Y -W 300 -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
rate-stonewall  = 810.555174
score           = 813.603723
t_delta         = 316.2440
t_end           = 2020-07-09 21:34:19

[find]
t_start         = 2020-07-09 21:34:20
exe             = mpirun --bind-to socket -np 320 -f /home/kalfizah/io500-new/hfile /home/kalfizah/mfu/install/bin/dfind --pool 9020212b-0f07-4d33-8f84-6ac45cb4d7eb --cont ce610953-295d-464c-af58-bd04279ded70 --svcl 1 --prefix /tmp/dfuse/ --stonewall 300 /tmp/dfuse/datafiles/2020.07.09-19.56.01-app -newer /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/timestampfile -size 3901 -name "*01*"
last-output     = "MATCHED 213024/493308279"
total-files     = 493308279
found           = 213024
score           = 1561.258730
t_delta         = 316.6142
t_end           = 2020-07-09 21:39:36

[ior-easy-read]
t_start         = 2020-07-09 21:39:36
exe             = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-easy/stonewall -t 256k -b 256g -r -R -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.prefix=/tmp/dfuse/
score           = 127.913005
t_delta         = 337.5071
t_end           = 2020-07-09 21:45:14

[mdtest-easy-stat]
t_start         = 2020-07-09 21:45:14
exe             = ./mdtest -n 4000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy-stonewall -T -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 4185.471912
t_delta         = 311.8157
t_end           = 2020-07-09 21:50:26

[ior-hard-read]
t_start         = 2020-07-09 21:50:26
exe             = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/file -O stoneWallingStatusFile=/tmp/dfuse/datafiles/2020.07.09-19.56.01-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1600000 -r -R -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.chunk_size=470080 --dfs.prefix=/tmp/dfuse/
score           = 80.002867
t_delta         = 418.9886
t_end           = 2020-07-09 21:57:25

[mdtest-hard-stat]
t_start         = 2020-07-09 21:57:25
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall -T -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 3430.205288
t_delta         = 78.0000
t_end           = 2020-07-09 21:58:43

[mdtest-easy-delete]
t_start         = 2020-07-09 21:58:44
exe             = ./mdtest -n 4000000 -u -L -F -P -N 1 -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-easy-stonewall -r -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 2249.345123
t_delta         = 592.0934
t_end           = 2020-07-09 22:08:36

[mdtest-hard-read]
t_start         = 2020-07-09 22:08:36
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall -E -X -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 1918.106511
t_delta         = 136.4002
t_end           = 2020-07-09 22:10:52

[mdtest-hard-delete]
t_start         = 2020-07-09 22:10:53
exe             = ./mdtest -n 1500000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard -x /tmp/dfuse/datafiles/2020.07.09-19.56.01-app/mdtest-hard-stonewall -r -a DFS --dfs.pool=9020212b-0f07-4d33-8f84-6ac45cb4d7eb --dfs.cont=ce610953-295d-464c-af58-bd04279ded70 --dfs.svcl=1 --dfs.oclass=S1 --dfs.prefix=/tmp/dfuse/
score           = 2188.869528
t_delta         = 734.5747
t_end           = 2020-07-09 22:23:07

[SCORE]
MD              = 2280.052351
BW              = 102.538088
SCORE           = 483.520638 
hash            = 5CC2B7F1
; END 2020-07-09 22:23:08
result_summary
IO500 version io500-isc20_v4-22
[RESULT]       ior-easy-write      101.038894 GiB/s  : time 426.716 seconds
[RESULT]    mdtest-easy-write     4240.894878 kIOPS : time 312.342 seconds
[RESULT]       ior-hard-write      106.913370 GiB/s  : time 314.964 seconds
[RESULT]    mdtest-hard-write      813.603723 kIOPS : time 316.244 seconds
[RESULT]                 find     1561.258730 kIOPS : time 316.614 seconds
[RESULT]        ior-easy-read      127.913005 GiB/s  : time 337.507 seconds
[RESULT]     mdtest-easy-stat     4185.471912 kIOPS : time 311.816 seconds
[RESULT]        ior-hard-read       80.002867 GiB/s  : time 418.989 seconds
[RESULT]     mdtest-hard-stat     3430.205288 kIOPS : time 78.000 seconds
[RESULT]   mdtest-easy-delete     2249.345123 kIOPS : time 592.093 seconds
[RESULT]     mdtest-hard-read     1918.106511 kIOPS : time 136.400 seconds
[RESULT]   mdtest-hard-delete     2188.869528 kIOPS : time 734.575 seconds
[SCORE] Bandwidth 102.538088 GB/s : IOPS 2280.052351 kiops : TOTAL 483.520638