[START TIME] 2019-05-09 06:23:54-04:00 [IOR EASY WRITE] time: 1557397434.947031486 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 9 06:23:56 2019 Command line : ./bin//ior --posix.odirect -t 1024k -b 25000m -F -w -C -Q 1 -g -G 27 -k -e -o ./10.18datafiles//ior_easy/ior_file_easy -O stoneWallingStatusFile=./10.18datafiles//ior_easy/stonewall -O stoneWallingWearOut=1 -D 300 Machine : Linux h10u23.int.janelia.org TestID : 0 StartTime : Thu May 9 06:23:56 2019 Path : /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles/ior_easy FS : 3623.2 TiB Used FS: 74.9% Inodes: 1312662.2 Mi Used Inodes: 56.6% Options: api : POSIX apiVersion : test filename : ./10.18datafiles//ior_easy/ior_file_easy access : file-per-process type : independent segments : 1 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 320 clients per node : 32 repetitions : 1 xfersize : 1 MiB blocksize : 24.41 GiB aggregate filesize : 7.63 TiB stonewallingTime : 300 stoneWallingWearOut : 1 Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- stonewalling pairs accessed min: 15369 max: 25000 -- min data: 15.0 GiB mean data: 20.2 GiB time: 300.0s write 19164 25600000 1024.00 0.561760 416.90 0.000265 417.46 0 Max Write: 19163.64 MiB/sec (20094.54 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum write 19163.64 19163.64 19163.64 0.00 19163.64 19163.64 19163.64 0.00 417.45712 0 320 32 1 1 1 1 0 0 1 26214400000 1048576 8000000.0 POSIX 0 Finished : Thu May 9 06:30:53 2019 [MDTEST EASY WRITE] time: 1557397854.156326068 -- started at 05/09/2019 06:30:54 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "80000" "-u" "-L" "-C" "-F" "-d" "./10.18datafiles//mdt_easy" "-x" "./10.18datafiles//mdt_easy-stonewall" "-W" "300" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.0% 320 tasks, 25600000 files Continue stonewall hit min: 55266 max: 80000 avg: 67014.2 SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 64825.618 64825.590 64825.604 0.008 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 1.817 1.817 1.817 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/09/2019 06:37:30 -- [CREATING TIMESTAMP] time: 1557398250.611402034 [IOR HARD WRITE] time: 1557398250.759909277 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 9 06:37:31 2019 Command line : ./bin//ior --posix.odirect -s 1800 -w -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o ./10.18datafiles//ior_hard/IOR_file -O stoneWallingStatusFile=./10.18datafiles//ior_hard/stonewall -O stoneWallingWearOut=1 -D 300 Machine : Linux h10u23.int.janelia.org TestID : 0 StartTime : Thu May 9 06:37:31 2019 Path : /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles/ior_hard FS : 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% Options: api : POSIX apiVersion : test filename : ./10.18datafiles//ior_hard/IOR_file access : single-shared-file type : independent segments : 1800 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 320 clients per node : 32 repetitions : 1 xfersize : 47008 bytes blocksize : 47008 bytes aggregate filesize : 25.22 GiB stonewallingTime : 300 stoneWallingWearOut : 1 Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- stonewalling pairs accessed min: 407 max: 541 -- min data: 0.0 GiB mean data: 0.0 GiB time: 300.7s WARNING: Expected aggregate file size = 27076608000. WARNING: Stat() of aggregate file size = 8138024960. WARNING: Using actual aggregate bytes moved = 8138024960. WARNING: maybe caused by deadlineForStonewalling write 24.15 45.91 45.91 0.015096 321.37 0.000308 321.38 0 Max Write: 24.15 MiB/sec (25.32 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum write 24.15 24.15 24.15 0.00 538.67 538.67 538.67 0.00 321.38353 0 320 32 1 0 1 1 0 0 1800 47008 47008 7761.0 POSIX 0 Finished : Thu May 9 06:42:53 2019 [MDTEST HARD WRITE] time: 1557398573.274868406 -- started at 05/09/2019 06:42:54 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "1800" "-C" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./10.18datafiles//mdt_hard" "-x" "./10.18datafiles//mdt_hard-stonewall" "-W" "300" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% 320 tasks, 576000 files Continue stonewall hit min: 758 max: 813 avg: 775.5 SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 826.634 826.634 826.634 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 513.630 513.630 513.630 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/09/2019 06:48:08 -- [PFIND EASY] time: 1557398888.911572553 Error stating file: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles//mdt_easy/#test-dir.0-0/m [0] rate: 1.230 kiops time: 128.8s found: 52 (scanned 158360 files, scanned dirents: 160331, unknown dirents: 72452) [3] rate: 1.263 kiops time: 128.8s found: 55 (scanned 162633 files, scanned dirents: 160000, unknown dirents: 68986) [1] rate: 1.992 kiops time: 128.8s found: 1022 (scanned 256425 files, scanned dirents: 260483, unknown dirents: 52659) [6] rate: 1.229 kiops time: 128.8s found: 19 (scanned 158294 files, scanned dirents: 160000, unknown dirents: 62164) [14] rate: 0.646 kiops time: 128.8s found: 18 (scanned 83240 files, scanned dirents: 80000, unknown dirents: 36573) [11] rate: 1.259 kiops time: 128.8s found: 23 (scanned 162109 files, scanned dirents: 160000, unknown dirents: 25082) [4] rate: 1.232 kiops time: 128.8s found: 53 (scanned 158629 files, scanned dirents: 160000, unknown dirents: 66992) [2] rate: 2.464 kiops time: 128.8s found: 2112 (scanned 317224 files, scanned dirents: 320000, unknown dirents: 73453) [5] rate: 1.233 kiops time: 128.8s found: 35 (scanned 158809 files, scanned dirents: 160000, unknown dirents: 66680) [16] rate: 0.021 kiops time: 128.8s found: 5 (scanned 2766 files, scanned dirents: 0, unknown dirents: 0) [7] rate: 1.242 kiops time: 128.8s found: 34 (scanned 159938 files, scanned dirents: 160000, unknown dirents: 59943) [19] rate: 0.666 kiops time: 128.8s found: 3 (scanned 85800 files, scanned dirents: 80000, unknown dirents: 22380) [9] rate: 1.249 kiops time: 128.8s found: 52 (scanned 160845 files, scanned dirents: 160000, unknown dirents: 45100) [15] rate: 1.228 kiops time: 128.8s found: 30 (scanned 158128 files, scanned dirents: 160000, unknown dirents: 70683) [20] rate: 0.633 kiops time: 128.8s found: 4 (scanned 81529 files, scanned dirents: 80000, unknown dirents: 23714) [23] rate: 0.667 kiops time: 128.8s found: 18 (scanned 85942 files, scanned dirents: 80000, unknown dirents: 12037) [8] rate: 1.236 kiops time: 128.8s found: 10 (scanned 159119 files, scanned dirents: 160000, unknown dirents: 47631) [25] rate: 0.639 kiops time: 128.8s found: 34 (scanned 82271 files, scanned dirents: 80000, unknown dirents: 31808) [10] rate: 0.651 kiops time: 128.8s found: 58 (scanned 83826 files, scanned dirents: 80000, unknown dirents: 25174) [61] rate: 1.231 kiops time: 128.8s found: 3 (scanned 158489 files, scanned dirents: 160000, unknown dirents: 29117) [22] rate: 0.650 kiops time: 128.8s found: 6 (scanned 83631 files, scanned dirents: 80000, unknown dirents: 13440) [21] rate: 2.448 kiops time: 128.8s found: 2 (scanned 315224 files, scanned dirents: 320000, unknown dirents: 94551) [13] rate: 1.253 kiops time: 128.8s found: 24 (scanned 161384 files, scanned dirents: 160000, unknown dirents: 52911) [18] rate: 1.844 kiops time: 128.8s found: 0 (scanned 237365 files, scanned dirents: 240000, unknown dirents: 41938) [12] rate: 0.087 kiops time: 128.8s found: 33 (scanned 11264 files, scanned dirents: 0, unknown dirents: 0) [29] rate: 1.833 kiops time: 128.8s found: 14 (scanned 235952 files, scanned dirents: 240000, unknown dirents: 80602) [30] rate: 1.240 kiops time: 128.8s found: 0 (scanned 159710 files, scanned dirents: 160000, unknown dirents: 45008) [52] rate: 0.679 kiops time: 128.8s found: 5 (scanned 87433 files, scanned dirents: 80000, unknown dirents: 22959) [17] rate: 1.253 kiops time: 128.8s found: 13 (scanned 161343 files, scanned dirents: 160000, unknown dirents: 53107) [50] rate: 1.831 kiops time: 128.8s found: 8 (scanned 235718 files, scanned dirents: 240000, unknown dirents: 77628) [26] rate: 1.831 kiops time: 128.8s found: 4 (scanned 235790 files, scanned dirents: 240000, unknown dirents: 43028) [27] rate: 0.072 kiops time: 128.8s found: 26 (scanned 9331 files, scanned dirents: 0, unknown dirents: 0) [54] rate: 1.223 kiops time: 128.8s found: 2 (scanned 157476 files, scanned dirents: 160000, unknown dirents: 50199) [24] rate: 1.257 kiops time: 128.8s found: 12 (scanned 161872 files, scanned dirents: 160000, unknown dirents: 44245) [35] rate: 1.229 kiops time: 128.8s found: 1 (scanned 158229 files, scanned dirents: 160000, unknown dirents: 69697) [60] rate: 0.668 kiops time: 128.8s found: 16 (scanned 85947 files, scanned dirents: 80000, unknown dirents: 22813) [28] rate: 1.232 kiops time: 128.8s found: 15 (scanned 158628 files, scanned dirents: 160000, unknown dirents: 67386) [37] rate: 0.084 kiops time: 128.8s found: 6 (scanned 10800 files, scanned dirents: 0, unknown dirents: 0) [31] rate: 1.849 kiops time: 128.8s found: 2 (scanned 238066 files, scanned dirents: 240000, unknown dirents: 101003) [38] rate: 1.836 kiops time: 128.8s found: 9 (scanned 236423 files, scanned dirents: 240000, unknown dirents: 79842) [46] rate: 1.244 kiops time: 128.8s found: 2 (scanned 160112 files, scanned dirents: 160000, unknown dirents: 47853) [57] rate: 0.643 kiops time: 128.8s found: 7 (scanned 82750 files, scanned dirents: 80000, unknown dirents: 22806) [32] rate: 3.039 kiops time: 128.8s found: 0 (scanned 391257 files, scanned dirents: 400000, unknown dirents: 72599) [62] rate: 0.041 kiops time: 128.8s found: 7 (scanned 5271 files, scanned dirents: 0, unknown dirents: 0) [33] rate: 0.121 kiops time: 128.8s found: 9 (scanned 15580 files, scanned dirents: 0, unknown dirents: 0) [49] rate: 0.085 kiops time: 128.8s found: 8 (scanned 11000 files, scanned dirents: 0, unknown dirents: 0) [36] rate: 3.648 kiops time: 128.8s found: 8 (scanned 469683 files, scanned dirents: 480000, unknown dirents: 101057) [51] rate: 1.847 kiops time: 128.8s found: 6 (scanned 237786 files, scanned dirents: 240000, unknown dirents: 41680) [45] rate: 1.285 kiops time: 128.8s found: 4 (scanned 165488 files, scanned dirents: 160000, unknown dirents: 48291) [63] rate: 1.841 kiops time: 128.8s found: 4 (scanned 237050 files, scanned dirents: 240000, unknown dirents: 102645) [39] rate: 1.850 kiops time: 128.8s found: 4 (scanned 238172 files, scanned dirents: 240000, unknown dirents: 64647) [55] rate: 0.631 kiops time: 128.8s found: 1 (scanned 81306 files, scanned dirents: 80000, unknown dirents: 30138) [42] rate: 2.454 kiops time: 128.8s found: 9 (scanned 315919 files, scanned dirents: 320000, unknown dirents: 78091) [56] rate: 1.231 kiops time: 128.8s found: 11 (scanned 158495 files, scanned dirents: 160000, unknown dirents: 40769) [43] rate: 0.657 kiops time: 128.8s found: 0 (scanned 84562 files, scanned dirents: 80000, unknown dirents: 27050) [34] rate: 0.642 kiops time: 128.8s found: 11 (scanned 82705 files, scanned dirents: 80000, unknown dirents: 36839) [41] rate: 1.853 kiops time: 128.8s found: 25 (scanned 238598 files, scanned dirents: 240000, unknown dirents: 62230) [53] rate: 0.035 kiops time: 128.8s found: 6 (scanned 4452 files, scanned dirents: 0, unknown dirents: 0) [44] rate: 1.838 kiops time: 128.8s found: 2 (scanned 236596 files, scanned dirents: 240000, unknown dirents: 38418) [48] rate: 2.482 kiops time: 128.8s found: 16 (scanned 319633 files, scanned dirents: 320000, unknown dirents: 60405) [58] rate: 1.235 kiops time: 128.8s found: 21 (scanned 159069 files, scanned dirents: 160000, unknown dirents: 38602) [40] rate: 1.483 kiops time: 128.8s found: 4 (scanned 190941 files, scanned dirents: 240000, unknown dirents: 77390) [47] rate: 0.318 kiops time: 128.8s found: 5 (scanned 40913 files, scanned dirents: 80000, unknown dirents: 34587) [59] rate: 1.260 kiops time: 128.8s found: 1 (scanned 162297 files, scanned dirents: 160000, unknown dirents: 42582) [64] rate: 3.041 kiops time: 128.8s found: 22 (scanned 391490 files, scanned dirents: 400000, unknown dirents: 83214) [65] rate: 1.293 kiops time: 128.8s found: 21 (scanned 166487 files, scanned dirents: 160000, unknown dirents: 32415) [68] rate: 1.228 kiops time: 128.8s found: 1 (scanned 158156 files, scanned dirents: 160000, unknown dirents: 35127) [69] rate: 1.258 kiops time: 128.8s found: 2 (scanned 162024 files, scanned dirents: 160000, unknown dirents: 38908) [75] rate: 0.637 kiops time: 128.8s found: 5 (scanned 81954 files, scanned dirents: 80000, unknown dirents: 20721) [77] rate: 1.239 kiops time: 128.8s found: 7 (scanned 159525 files, scanned dirents: 160000, unknown dirents: 30520) [78] rate: 1.846 kiops time: 128.8s found: 4 (scanned 237698 files, scanned dirents: 240000, unknown dirents: 73709) [82] rate: 2.447 kiops time: 128.8s found: 0 (scanned 315076 files, scanned dirents: 320000, unknown dirents: 85319) [79] rate: 0.637 kiops time: 128.8s found: 4 (scanned 81964 files, scanned dirents: 80000, unknown dirents: 35698) [86] rate: 1.278 kiops time: 128.8s found: 8 (scanned 164594 files, scanned dirents: 160000, unknown dirents: 33697) [66] rate: 1.246 kiops time: 128.8s found: 7 (scanned 160418 files, scanned dirents: 160000, unknown dirents: 39713) [85] rate: 2.444 kiops time: 128.8s found: 2 (scanned 314703 files, scanned dirents: 320000, unknown dirents: 49406) [67] rate: 0.645 kiops time: 128.8s found: 9 (scanned 83054 files, scanned dirents: 80000, unknown dirents: 26072) [89] rate: 1.241 kiops time: 128.8s found: 9 (scanned 159753 files, scanned dirents: 160000, unknown dirents: 40154) [72] rate: 1.231 kiops time: 128.8s found: 8 (scanned 158506 files, scanned dirents: 160000, unknown dirents: 62949) [83] rate: 0.650 kiops time: 128.8s found: 6 (scanned 83729 files, scanned dirents: 80000, unknown dirents: 35840) [74] rate: 1.230 kiops time: 128.8s found: 4 (scanned 158427 files, scanned dirents: 160000, unknown dirents: 38750) [84] rate: 1.234 kiops time: 128.8s found: 1 (scanned 158945 files, scanned dirents: 160000, unknown dirents: 54608) [70] rate: 0.644 kiops time: 128.8s found: 0 (scanned 82913 files, scanned dirents: 80000, unknown dirents: 14841) [73] rate: 1.234 kiops time: 128.8s found: 13 (scanned 158902 files, scanned dirents: 160000, unknown dirents: 54770) [71] rate: 1.234 kiops time: 128.8s found: 0 (scanned 158901 files, scanned dirents: 160000, unknown dirents: 68003) [96] rate: 1.842 kiops time: 128.8s found: 0 (scanned 237166 files, scanned dirents: 240000, unknown dirents: 46035) [76] rate: 0.633 kiops time: 128.8s found: 0 (scanned 81565 files, scanned dirents: 80000, unknown dirents: 21157) [90] rate: 0.659 kiops time: 128.8s found: 6 (scanned 84836 files, scanned dirents: 80000, unknown dirents: 9027) [102] rate: 1.863 kiops time: 128.8s found: 12 (scanned 239828 files, scanned dirents: 240000, unknown dirents: 41035) [81] rate: 1.843 kiops time: 128.8s found: 15 (scanned 237279 files, scanned dirents: 240000, unknown dirents: 83172) [98] rate: 3.041 kiops time: 128.8s found: 5 (scanned 391542 files, scanned dirents: 400000, unknown dirents: 93838) [114] rate: 1.834 kiops time: 128.8s found: 0 (scanned 236163 files, scanned dirents: 240000, unknown dirents: 87843) [94] rate: 1.845 kiops time: 128.8s found: 0 (scanned 237514 files, scanned dirents: 240000, unknown dirents: 97340) [117] rate: 1.831 kiops time: 128.8s found: 2 (scanned 235746 files, scanned dirents: 240000, unknown dirents: 83999) [80] rate: 0.644 kiops time: 128.8s found: 11 (scanned 82937 files, scanned dirents: 80000, unknown dirents: 35096) [100] rate: 1.247 kiops time: 128.8s found: 7 (scanned 160584 files, scanned dirents: 160000, unknown dirents: 46298) [87] rate: 0.094 kiops time: 128.8s found: 14 (scanned 12150 files, scanned dirents: 0, unknown dirents: 0) [107] rate: 0.709 kiops time: 128.8s found: 20 (scanned 91280 files, scanned dirents: 80000, unknown dirents: 18188) [119] rate: 2.436 kiops time: 128.8s found: 1 (scanned 313666 files, scanned dirents: 320000, unknown dirents: 73056) [105] rate: 1.257 kiops time: 128.8s found: 24 (scanned 161867 files, scanned dirents: 160000, unknown dirents: 37577) [116] rate: 1.238 kiops time: 128.8s found: 3 (scanned 159444 files, scanned dirents: 160000, unknown dirents: 65588) [104] rate: 1.839 kiops time: 128.8s found: 1 (scanned 236794 files, scanned dirents: 240000, unknown dirents: 59313) [95] rate: 1.834 kiops time: 128.8s found: 0 (scanned 236135 files, scanned dirents: 240000, unknown dirents: 102561) [101] rate: 1.252 kiops time: 128.8s found: 2 (scanned 161219 files, scanned dirents: 160000, unknown dirents: 42725) [88] rate: 1.842 kiops time: 128.8s found: 10 (scanned 237159 files, scanned dirents: 240000, unknown dirents: 61977) [111] rate: 1.243 kiops time: 128.8s found: 44 (scanned 160035 files, scanned dirents: 160000, unknown dirents: 65502) [91] rate: 1.268 kiops time: 128.8s found: 8 (scanned 163243 files, scanned dirents: 160000, unknown dirents: 65056) [99] rate: 1.268 kiops time: 128.8s found: 6 (scanned 163202 files, scanned dirents: 160000, unknown dirents: 47713) [93] rate: 0.838 kiops time: 128.8s found: 11 (scanned 107884 files, scanned dirents: 80000, unknown dirents: 22031) [97] rate: 0.074 kiops time: 128.8s found: 62 (scanned 9550 files, scanned dirents: 0, unknown dirents: 0) [108] rate: 1.846 kiops time: 128.8s found: 1 (scanned 237715 files, scanned dirents: 240000, unknown dirents: 44364) [112] rate: 1.836 kiops time: 128.8s found: 2 (scanned 236366 files, scanned dirents: 240000, unknown dirents: 44444) [130] rate: 0.100 kiops time: 128.8s found: 32 (scanned 12902 files, scanned dirents: 0, unknown dirents: 0) [118] rate: 1.237 kiops time: 128.8s found: 17 (scanned 159281 files, scanned dirents: 160000, unknown dirents: 70148) [92] rate: 0.866 kiops time: 128.8s found: 15 (scanned 111500 files, scanned dirents: 160000, unknown dirents: 55772) [149] rate: 1.831 kiops time: 128.8s found: 7 (scanned 235768 files, scanned dirents: 240000, unknown dirents: 77335) [103] rate: 0.672 kiops time: 128.8s found: 8 (scanned 86499 files, scanned dirents: 80000, unknown dirents: 23775) [134] rate: 1.263 kiops time: 128.8s found: 10 (scanned 162682 files, scanned dirents: 160000, unknown dirents: 31512) [115] rate: 1.836 kiops time: 128.8s found: 6 (scanned 236351 files, scanned dirents: 240000, unknown dirents: 78480) [110] rate: 1.233 kiops time: 128.8s found: 21 (scanned 158754 files, scanned dirents: 160000, unknown dirents: 49235) [132] rate: 1.248 kiops time: 128.8s found: 16 (scanned 160628 files, scanned dirents: 160000, unknown dirents: 48698) [123] rate: 1.229 kiops time: 128.8s found: 3 (scanned 158192 files, scanned dirents: 160000, unknown dirents: 64099) [109] rate: 0.074 kiops time: 128.8s found: 21 (scanned 9517 files, scanned dirents: 0, unknown dirents: 0) [133] rate: 1.848 kiops time: 128.8s found: 6 (scanned 237892 files, scanned dirents: 240000, unknown dirents: 60668) [113] rate: 0.070 kiops time: 128.8s found: 23 (scanned 9028 files, scanned dirents: 0, unknown dirents: 0) [106] rate: 0.637 kiops time: 128.8s found: 26 (scanned 81990 files, scanned dirents: 80000, unknown dirents: 16263) [135] rate: 1.874 kiops time: 128.8s found: 12 (scanned 241344 files, scanned dirents: 240000, unknown dirents: 48422) [126] rate: 1.271 kiops time: 128.8s found: 1 (scanned 163712 files, scanned dirents: 160000, unknown dirents: 48093) [145] rate: 1.239 kiops time: 128.8s found: 0 (scanned 159528 files, scanned dirents: 160000, unknown dirents: 61015) [128] rate: 1.235 kiops time: 128.8s found: 6 (scanned 158956 files, scanned dirents: 160000, unknown dirents: 42499) [124] rate: 1.227 kiops time: 128.8s found: 8 (scanned 157980 files, scanned dirents: 160000, unknown dirents: 55813) [147] rate: 1.846 kiops time: 128.8s found: 9 (scanned 237650 files, scanned dirents: 240000, unknown dirents: 58115) [131] rate: 0.662 kiops time: 128.8s found: 35 (scanned 85219 files, scanned dirents: 80000, unknown dirents: 37330) [129] rate: 1.280 kiops time: 128.8s found: 14 (scanned 164824 files, scanned dirents: 160000, unknown dirents: 27950) [125] rate: 1.841 kiops time: 128.8s found: 4 (scanned 237022 files, scanned dirents: 240000, unknown dirents: 37705) [142] rate: 1.840 kiops time: 128.8s found: 30 (scanned 236963 files, scanned dirents: 240000, unknown dirents: 73022) [121] rate: 0.058 kiops time: 128.8s found: 8 (scanned 7518 files, scanned dirents: 0, unknown dirents: 0) [138] rate: 1.245 kiops time: 128.8s found: 6 (scanned 160294 files, scanned dirents: 160000, unknown dirents: 34482) [151] rate: 0.641 kiops time: 128.8s found: 9 (scanned 82531 files, scanned dirents: 80000, unknown dirents: 20259) [120] rate: 1.258 kiops time: 128.8s found: 10 (scanned 162016 files, scanned dirents: 160000, unknown dirents: 23684) [122] rate: 1.225 kiops time: 128.8s found: 1 (scanned 157756 files, scanned dirents: 160000, unknown dirents: 66620) [139] rate: 0.068 kiops time: 128.8s found: 33 (scanned 8737 files, scanned dirents: 0, unknown dirents: 0) [127] rate: 1.845 kiops time: 128.8s found: 4 (scanned 237527 files, scanned dirents: 240000, unknown dirents: 99233) [137] rate: 2.459 kiops time: 128.8s found: 11 (scanned 316621 files, scanned dirents: 320000, unknown dirents: 100900) [144] rate: 1.834 kiops time: 128.8s found: 1 (scanned 236142 files, scanned dirents: 240000, unknown dirents: 88265) [148] rate: 0.650 kiops time: 128.8s found: 12 (scanned 83665 files, scanned dirents: 80000, unknown dirents: 36525) [141] rate: 0.679 kiops time: 128.8s found: 12 (scanned 87412 files, scanned dirents: 80000, unknown dirents: 25753) [146] rate: 1.236 kiops time: 128.8s found: 9 (scanned 159081 files, scanned dirents: 160000, unknown dirents: 55393) [143] rate: 1.245 kiops time: 128.8s found: 2 (scanned 160334 files, scanned dirents: 160000, unknown dirents: 69740) [150] rate: 1.840 kiops time: 128.8s found: 2 (scanned 236904 files, scanned dirents: 240000, unknown dirents: 68538) [152] rate: 0.638 kiops time: 128.8s found: 1 (scanned 82099 files, scanned dirents: 80000, unknown dirents: 29810) [155] rate: 0.647 kiops time: 128.8s found: 14 (scanned 83255 files, scanned dirents: 80000, unknown dirents: 10363) [153] rate: 1.230 kiops time: 128.8s found: 18 (scanned 158348 files, scanned dirents: 160000, unknown dirents: 52329) [136] rate: 0.683 kiops time: 128.8s found: 39 (scanned 87893 files, scanned dirents: 80000, unknown dirents: 7865) [154] rate: 1.850 kiops time: 128.8s found: 12 (scanned 238202 files, scanned dirents: 240000, unknown dirents: 45718) [140] rate: 2.447 kiops time: 128.8s found: 3 (scanned 315016 files, scanned dirents: 320000, unknown dirents: 62230) [156] rate: 1.253 kiops time: 128.8s found: 11 (scanned 161323 files, scanned dirents: 160000, unknown dirents: 47034) [158] rate: 1.260 kiops time: 128.8s found: 5 (scanned 162223 files, scanned dirents: 160000, unknown dirents: 50898) [157] rate: 2.440 kiops time: 128.8s found: 0 (scanned 314225 files, scanned dirents: 320000, unknown dirents: 81079) [159] rate: 0.633 kiops time: 128.8s found: 12 (scanned 81495 files, scanned dirents: 80000, unknown dirents: 36953) [DONE] rate: 202.087 kiops time: 128.8s found: 4974 (scanned 26020484 files, scanned dirents: 26020814, unknown dirents: 7450653) MATCHED 4974/26020484 time: 1557399018.504817480 [IOR EASY READ] time: 1557399018.506227779 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 9 06:50:19 2019 Command line : ./bin//ior --posix.odirect -t 1024k -b 25000m -F -r -R -C -Q 1 -g -G 27 -k -e -o ./10.18datafiles//ior_easy/ior_file_easy -O stoneWallingStatusFile=./10.18datafiles//ior_easy/stonewall Machine : Linux h10u23.int.janelia.org TestID : 0 StartTime : Thu May 9 06:50:19 2019 Path : /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles/ior_easy FS : 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% Options: api : POSIX apiVersion : test filename : ./10.18datafiles//ior_easy/ior_file_easy access : file-per-process type : independent segments : 1 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 320 clients per node : 32 repetitions : 1 xfersize : 1 MiB blocksize : 24.41 GiB aggregate filesize : 7.63 TiB Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- read 15613 25600000 1024.00 0.023653 512.36 0.000786 512.39 0 Max Read: 15613.15 MiB/sec (16371.58 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum read 15613.15 15613.15 15613.15 0.00 15613.15 15613.15 15613.15 0.00 512.38851 0 320 32 1 1 1 1 0 0 1 26214400000 1048576 8000000.0 POSIX 0 Finished : Thu May 9 06:58:51 2019 [MDTEST EASY STAT] time: 1557399531.894430334 -- started at 05/09/2019 06:58:52 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "80000" "-u" "-L" "-T" "-F" "-d" "./10.18datafiles//mdt_easy" "-x" "./10.18datafiles//mdt_easy-stonewall" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% 320 tasks, 25600000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 307947.260 307946.871 307947.016 0.089 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/09/2019 07:00:15 -- [IOR HARD READ] time: 1557399615.934118294 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 9 07:00:16 2019 Command line : ./bin//ior --posix.odirect -s 1800 -r -R -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -s 1800 --posix.odirect -o ./10.18datafiles//ior_hard/IOR_file -O stoneWallingStatusFile=./10.18datafiles//ior_hard/stonewall Machine : Linux h10u23.int.janelia.org TestID : 0 StartTime : Thu May 9 07:00:16 2019 Path : /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles/ior_hard FS : 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% Options: api : POSIX apiVersion : test filename : ./10.18datafiles//ior_hard/IOR_file access : single-shared-file type : independent segments : 1800 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 320 clients per node : 32 repetitions : 1 xfersize : 47008 bytes blocksize : 47008 bytes aggregate filesize : 25.22 GiB Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- WARNING: Expected aggregate file size = 27076608000. WARNING: Stat() of aggregate file size = 8138024960. WARNING: Using actual aggregate bytes moved = 8138024960. read 4820 45.91 45.91 0.033297 1.58 0.001790 1.61 0 Max Read: 4819.72 MiB/sec (5053.84 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum read 4819.72 4819.72 4819.72 0.00 107510.32 107510.32 107510.32 0.00 1.61026 0 320 32 1 0 1 1 0 0 1800 47008 47008 7761.0 POSIX 0 Finished : Thu May 9 07:00:18 2019 [MDTEST HARD STAT] time: 1557399618.501274396 -- started at 05/09/2019 07:00:18 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "1800" "-T" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./10.18datafiles//mdt_hard" "-x" "./10.18datafiles//mdt_hard-stonewall" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% 320 tasks, 576000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 277740.409 271302.115 277714.826 359.057 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/09/2019 07:00:19 -- [MDTEST EASY DELETE] time: 1557399619.904169514 -- started at 05/09/2019 07:00:20 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "80000" "-u" "-L" "-r" "-F" "-d" "./10.18datafiles//mdt_easy" "-x" "./10.18datafiles//mdt_easy-stonewall" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.1% 320 tasks, 25600000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 48604.461 48604.443 48604.451 0.006 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 1.781 1.781 1.781 0.000 -- finished at 05/09/2019 07:09:08 -- [MDTEST HARD READ] time: 1557400148.634080376 -- started at 05/09/2019 07:09:09 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "1800" "-E" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./10.18datafiles//mdt_hard" "-x" "./10.18datafiles//mdt_hard-stonewall" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.0% 320 tasks, 576000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 137741.609 137739.192 137740.366 0.730 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/09/2019 07:09:11 -- [MDTEST HARD DELETE] time: 1557400151.473663869 -- started at 05/09/2019 07:09:11 -- mdtest-1.9.3 was launched with 320 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "1800" "-r" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./10.18datafiles//mdt_hard" "-x" "./10.18datafiles//mdt_hard-stonewall" Path: /groups/scicomp/scicompsys/io-500-update/io-500-script/10.18datafiles FS: 3623.2 TiB Used FS: 75.1% Inodes: 1312662.2 Mi Used Inodes: 58.0% 320 tasks, 576000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 419.972 419.972 419.972 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 9.163 9.163 9.163 0.000 -- finished at 05/09/2019 07:19:31 -- Deleting IO-500 ./10.18datafiles/ time: 1557400771.631996049 [END TIME] 2019-05-09 07:20:46-04:00 [IO-500 COMPLETED] Result also in config-10_32-result-1557397434.txt; Now use io-500-score.sh to compute the score