- ior-easy-read
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Wed Jul 8 18:16:33 2020
Command line : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine : Linux o03
TestID : 0
StartTime : Wed Jul 8 18:16:33 2020
Path : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy
FS : 428.3 TiB Used FS: 14.0% Inodes: 75.8 Mi Used Inodes: 100.0%
Options:
api : CEPHFS
apiVersion :
test filename : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/ior_file_easy
access : file-per-process
type : independent
segments : 1
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 8
tasks : 256
clients per node : 32
repetitions : 1
xfersize : 2 MiB
blocksize : 9.46 TiB
aggregate filesize : 2421.88 TiB
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
WARNING: Expected aggregate file size = 2662879723520000.
WARNING: Stat() of aggregate file size = 29182155292672.
WARNING: Using actual aggregate bytes moved = 29182155292672.
read 72594 36330 0.000077 10158080000 2048.00 0.353169 383.02 0.001075 383.37 0
Max Read: 72593.55 MiB/sec (76119.86 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
read 72593.55 72593.55 72593.55 0.00 36296.78 36296.78 36296.78 0.00 383.37112 NA NA 0 256 32 1 1 1 1 0 0 1 10401873920000 2097152 27830272.0 CEPHFS 0
Finished : Wed Jul 8 18:22:57 2020
- ior-easy-write
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Wed Jul 8 17:48:16 2020
Command line : ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine : Linux o03
TestID : 0
StartTime : Wed Jul 8 17:48:16 2020
Path : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy
FS : 429.8 TiB Used FS: 7.0% Inodes: 7.5 Mi Used Inodes: 100.0%
Options:
api : CEPHFS
apiVersion :
test filename : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/ior_file_easy
access : file-per-process
type : independent
segments : 1
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 8
tasks : 256
clients per node : 32
repetitions : 1
xfersize : 2 MiB
blocksize : 9.46 TiB
aggregate filesize : 2421.88 TiB
stonewallingTime : 300
stoneWallingWearOut : 1
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
stonewalling pairs accessed min: 24162 max: 54356 -- min data: 47.2 GiB mean data: 80.2 GiB time: 301.5s
WARNING: Expected aggregate file size = 2662879723520000.
WARNING: Stat() of aggregate file size = 29182155292672.
WARNING: Using actual aggregate bytes moved = 29182155292672.
WARNING: maybe caused by deadlineForStonewalling
write 59254 29629 0.000070 10158080000 2048.00 0.028567 469.65 0.000449 469.68 0
Max Write: 59254.27 MiB/sec (62132.61 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 59254.27 59254.27 59254.27 0.00 29627.14 29627.14 29627.14 0.00 469.67535 301.49 69704.21 0 256 32 1 1 1 1 0 0 1 10401873920000 2097152 27830272.0 CEPHFS 0
Finished : Wed Jul 8 17:56:06 2020
- ior-hard-read
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Wed Jul 8 18:25:00 2020
Command line : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine : Linux o03
TestID : 0
StartTime : Wed Jul 8 18:25:00 2020
Path : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard
FS : 428.3 TiB Used FS: 14.0% Inodes: 75.8 Mi Used Inodes: 100.0%
Options:
api : CEPHFS
apiVersion :
test filename : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/file
access : single-shared-file
type : independent
segments : 1000000
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 8
tasks : 256
clients per node : 32
repetitions : 1
xfersize : 47008 bytes
blocksize : 47008 bytes
aggregate filesize : 10.94 TiB
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
stonewalling pairs accessed min: 304385 max: 304385 -- min data: 13.3 GiB mean data: 13.3 GiB time: 242.1s
WARNING: Expected aggregate file size = 12034048000000.
WARNING: Stat() of aggregate file size = 3662983700480.
WARNING: Using actual aggregate bytes moved = 3662983700480.
read 14420 321832 242.12 45.91 45.91 0.139184 242.12 0.001834 242.26 0
Max Read: 14419.55 MiB/sec (15119.99 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
read 14419.55 14419.55 14419.55 0.00 321647.18 321647.18 321647.18 0.00 242.26098 242.11 14428.42 0 256 32 1 0 1 1 0 0 1000000 47008 47008 3493293.5 CEPHFS 0
Finished : Wed Jul 8 18:29:03 2020
- ior-hard-write
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Wed Jul 8 18:05:32 2020
Command line : ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -w -D 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
Machine : Linux o03
TestID : 0
StartTime : Wed Jul 8 18:05:32 2020
Path : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard
FS : 428.5 TiB Used FS: 13.2% Inodes: 69.9 Mi Used Inodes: 100.0%
Options:
api : CEPHFS
apiVersion :
test filename : /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/file
access : single-shared-file
type : independent
segments : 1000000
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 8
tasks : 256
clients per node : 32
repetitions : 1
xfersize : 47008 bytes
blocksize : 47008 bytes
aggregate filesize : 10.94 TiB
stonewallingTime : 300
stoneWallingWearOut : 1
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
stonewalling pairs accessed min: 303763 max: 304385 -- min data: 13.3 GiB mean data: 13.3 GiB time: 300.2s
WARNING: Expected aggregate file size = 12034048000000.
WARNING: Stat() of aggregate file size = 3662983700480.
WARNING: Using actual aggregate bytes moved = 3662983700480.
WARNING: maybe caused by deadlineForStonewalling
write 11595 258953 300.64 45.91 45.91 0.352667 300.91 0.000299 301.27 0
Max Write: 11595.36 MiB/sec (12158.61 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 11595.36 11595.36 11595.36 0.00 258649.86 258649.86 258649.86 0.00 301.26659 300.21 11616.82 0 256 32 1 0 1 1 0 0 1000000 47008 47008 3493293.5 CEPHFS 0
Finished : Wed Jul 8 18:10:34 2020
- mdtest-easy-delete
-
-- started at 07/08/2020 18:29:27 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '240000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy-stonewall' '-r' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.3 TiB Used FS: 14.0% Inodes: 0.0 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 61440000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 116075.472 116075.218 116075.346 0.060
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 7.139 7.139 7.139 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 529.312 529.311 529.311 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.140 0.140 0.140 0.000
-- finished at 07/08/2020 18:38:19 --
- mdtest-easy-stat
-
-- started at 07/08/2020 18:22:57 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '240000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy-stonewall' '-T' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.3 TiB Used FS: 14.0% Inodes: 0.0 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 61440000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 501629.721 501623.574 501626.684 1.220
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 122.482 122.481 122.482 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/08/2020 18:24:59 --
- mdtest-easy-write
-
-- started at 07/08/2020 17:56:06 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '240000' '-u' '-L' '-F' '-P' '-N' '1' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy-stonewall' '-C' '-Y' '-W' '300' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.6 TiB Used FS: 13.2% Inodes: 0.0 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 61440000 files
Continue stonewall hit min: 154805 max: 240000 avg: 234139.8
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 108678.990 108678.892 108678.951 0.022
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 199687.399 NA
Tree creation : 21.681 21.681 21.681 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 565.335 565.335 565.335 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 300.168 NA
Tree creation : 0.046 0.046 0.046 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/08/2020 18:05:32 --
- mdtest-hard-delete
-
-- started at 07/08/2020 18:39:13 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall' '-r' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.3 TiB Used FS: 14.0% Inodes: 54.8 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 25600000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 36327.352 36326.942 36327.136 0.098
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 6.504 6.504 6.504 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 84.763 84.762 84.762 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.154 0.154 0.154 0.000
-- finished at 07/08/2020 18:40:38 --
- mdtest-hard-read
-
-- started at 07/08/2020 18:38:49 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall' '-E' '-X' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.3 TiB Used FS: 14.0% Inodes: 54.8 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 25600000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 139259.902 139252.824 139256.253 1.639
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 22.112 22.111 22.112 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/08/2020 18:39:12 --
- mdtest-hard-stat
-
-- started at 07/08/2020 18:29:03 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall' '-T' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.3 TiB Used FS: 14.0% Inodes: 0.0 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 25600000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 127888.103 127884.534 127886.467 0.789
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 24.078 24.077 24.077 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/08/2020 18:29:27 --
- mdtest-hard-write
-
-- started at 07/08/2020 18:10:34 --
mdtest-3.3.0+dev was launched with 256 total task(s) on 8 node(s)
Command line used: ./mdtest '-n' '100000' '-t' '-w' '3901' '-e' '3901' '-P' '-N' '1' '-F' '-d' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard' '-x' '/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall' '-C' '-Y' '-W' '300' '-a' 'CEPHFS' '--cephfs.user=admin' '--cephfs.conf=/etc/ceph/ceph.conf' '--cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0'
Path: /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app
FS: 428.3 TiB Used FS: 14.0% Inodes: 0.0 Mi Used Inodes: 100.0%
Nodemap: 1111111111111111111111111111111100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 32 for each phase.
256 tasks, 25600000 files
Continue stonewall hit min: 9006 max: 12028 avg: 10452.5
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 9873.110 9873.105 9873.107 0.001
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 8921.469 NA
Tree creation : 9.001 9.001 9.001 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 311.874 311.874 311.874 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 299.932 NA
Tree creation : 0.111 0.111 0.111 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/08/2020 18:15:46 --
- result
-
version = io500-isc20_v4
config-hash = 27ECB143
result-dir = ./results/2020.07.08-16.53.40-app
; START 2020-07-08 17:48:16
[ior-easy-write]
t_start = 2020-07-08 17:48:16
exe = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/stonewall -t 2m -b 9920000m -F -w -D 300 -O stoneWallingWearOut=1 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
throughput-stonewall = 11185.87
score = 57.865502
t_delta = 469.9090
t_end = 2020-07-08 17:56:06
[mdtest-easy-write]
t_start = 2020-07-08 17:56:06
exe = ./mdtest -n 240000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy-stonewall -C -Y -W 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
rate-stonewall = 199.687399
score = 108.678956
t_delta = 565.7802
t_end = 2020-07-08 18:05:32
[timestamp]
t_start = 2020-07-08 18:05:32
t_delta = 0.0053
t_end = 2020-07-08 18:05:32
[ior-hard-write]
t_start = 2020-07-08 18:05:32
exe = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -w -D 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
throughput-stonewall = 2894.04
score = 11.323590
t_delta = 301.7669
t_end = 2020-07-08 18:10:34
[mdtest-hard-write]
t_start = 2020-07-08 18:10:34
exe = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall -C -Y -W 300 -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
rate-stonewall = 8.921469
score = 9.873107
t_delta = 312.5518
t_end = 2020-07-08 18:15:46
[find]
t_start = 2020-07-08 18:15:46
exe = ./pfind /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app -newer /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/timestampfile -size 3901c -name *01* -C -q 10000
found = 110162
total-files = 64519430
score = 1394.901467
t_delta = 46.2558
t_end = 2020-07-08 18:16:33
[ior-easy-read]
t_start = 2020-07-08 18:16:33
exe = ./ior -C -Q 1 -g -G 271 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/ior_file_easy -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-easy/stonewall -t 2m -b 9920000m -F -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 70.892143
t_delta = 384.0279
t_end = 2020-07-08 18:22:57
[mdtest-easy-stat]
t_start = 2020-07-08 18:22:57
exe = ./mdtest -n 240000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy-stonewall -T -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 501.629537
t_delta = 122.8602
t_end = 2020-07-08 18:25:00
[ior-hard-read]
t_start = 2020-07-08 18:25:00
exe = ./ior -C -Q 1 -g -G 27 -k -e -o /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/file -O stoneWallingStatusFile=/tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/ior-hard/stonewall -O stoneWallingWearOut=1 -t 47008 -b 47008 -s 1000000 -r -R -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 14.081589
t_delta = 242.8458
t_end = 2020-07-08 18:29:03
[mdtest-hard-stat]
t_start = 2020-07-08 18:29:03
exe = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall -T -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 127.887913
t_delta = 24.4290
t_end = 2020-07-08 18:29:27
[mdtest-easy-delete]
t_start = 2020-07-08 18:29:27
exe = ./mdtest -n 240000 -u -L -F -P -N 1 -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-easy-stonewall -r -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 116.075434
t_delta = 562.2122
t_end = 2020-07-08 18:38:49
[mdtest-hard-read]
t_start = 2020-07-08 18:38:49
exe = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall -E -X -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 139.253021
t_delta = 22.5978
t_end = 2020-07-08 18:39:12
[mdtest-hard-delete]
t_start = 2020-07-08 18:39:12
exe = ./mdtest -n 100000 -t -w 3901 -e 3901 -P -N 1 -F -d /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard -x /tmp/cbt/mnt/cbt-cephfs-kernel/0/io500/io500-app/datafiles/2020.07.08-16.53.40-app/mdtest-hard-stonewall -r -a CEPHFS --cephfs.user=admin --cephfs.conf=/etc/ceph/ceph.conf --cephfs.prefix=/tmp/cbt/mnt/cbt-cephfs-kernel/0
score = 36.327299
t_delta = 85.2415
t_end = 2020-07-08 18:40:38
[SCORE]
MD = 124.134267
BW = 28.438940
SCORE = 59.415882
hash = E9426B44
; END 2020-07-08 18:41:18
- result_summary
-
IO500 version io500-isc20_v4
[RESULT] ior-easy-write 57.865502 GiB/s : time 469.909 seconds
[RESULT] mdtest-easy-write 108.678956 kIOPS : time 565.780 seconds
[RESULT] ior-hard-write 11.323590 GiB/s : time 301.767 seconds
[RESULT] mdtest-hard-write 9.873107 kIOPS : time 312.552 seconds
[RESULT] find 1394.901467 kIOPS : time 46.256 seconds
[RESULT] ior-easy-read 70.892143 GiB/s : time 384.028 seconds
[RESULT] mdtest-easy-stat 501.629537 kIOPS : time 122.860 seconds
[RESULT] ior-hard-read 14.081589 GiB/s : time 242.846 seconds
[RESULT] mdtest-hard-stat 127.887913 kIOPS : time 24.429 seconds
[RESULT] mdtest-easy-delete 116.075434 kIOPS : time 562.212 seconds
[RESULT] mdtest-hard-read 139.253021 kIOPS : time 22.598 seconds
[RESULT] mdtest-hard-delete 36.327299 kIOPS : time 85.242 seconds
[SCORE] Bandwidth 28.438940 GB/s : IOPS 124.134267 kiops : TOTAL 59.415882