[START TIME] 2019-05-16 17:29:24-04:00 [IOR EASY WRITE] time: 1558042164.788230191 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 16 17:29:35 2019 Command line : ./bin//ior --posix.odirect -t 1m -b 6000m -F -w -C -Q 1 -g -G 27 -k -e -o ./io500-datadir_10gpu//ior_easy/ior_file_easy -O stoneWallingStatusFile=./io500-datadir_10gpu//ior_easy/stonewall -O stoneWallingWearOut=1 -D 300 Machine : Linux f16u15.int.janelia.org TestID : 0 StartTime : Thu May 16 17:29:35 2019 Path : /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu/ior_easy FS : 100.0 TiB Used FS: 43.8% Inodes: 637.8 Mi Used Inodes: 11.1% Options: api : POSIX apiVersion : test filename : ./io500-datadir_10gpu//ior_easy/ior_file_easy access : file-per-process type : independent segments : 1 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 3200 clients per node : 320 repetitions : 1 xfersize : 1 MiB blocksize : 5.86 GiB aggregate filesize : 18.31 TiB stonewallingTime : 300 stoneWallingWearOut : 1 Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- stonewalling pairs accessed min: 2193 max: 6000 -- min data: 2.1 GiB mean data: 5.2 GiB time: 300.7s write 43514 6144000 1024.00 0.863808 440.36 0.027879 441.24 0 Max Write: 43514.12 MiB/sec (45627.86 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum write 43514.12 43514.12 43514.12 0.00 43514.12 43514.12 43514.12 0.00 441.23608 0 3200 320 1 1 1 1 0 0 1 6291456000 1048576 19200000.0 POSIX 0 Finished : Thu May 16 17:36:56 2019 [MDTEST EASY WRITE] time: 1558042617.312197362 -- started at 05/16/2019 17:37:04 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "15000" "-u" "-L" "-C" "-F" "-d" "./io500-datadir_10gpu//mdt_easy" "-x" "./io500-datadir_10gpu//mdt_easy-stonewall" "-W" "300" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 62.1% Inodes: 637.8 Mi Used Inodes: 14.0% 3200 tasks, 48000000 files Continue stonewall hit min: 7065 max: 15000 avg: 13597.3 SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 105670.345 105668.176 105669.911 0.436 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.813 0.813 0.813 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/16/2019 17:44:40 -- [CREATING TIMESTAMP] time: 1558043080.416416005 [IOR HARD WRITE] time: 1558043080.424034906 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 16 17:44:47 2019 Command line : ./bin//ior --posix.odirect -s 22000 -w -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o ./io500-datadir_10gpu//ior_hard/IOR_file -O stoneWallingStatusFile=./io500-datadir_10gpu//ior_hard/stonewall -O stoneWallingWearOut=1 -D 300 Machine : Linux f16u15.int.janelia.org TestID : 0 StartTime : Thu May 16 17:44:47 2019 Path : /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu/ior_hard FS : 100.0 TiB Used FS: 62.3% Inodes: 637.8 Mi Used Inodes: 22.1% Options: api : POSIX apiVersion : test filename : ./io500-datadir_10gpu//ior_hard/IOR_file access : single-shared-file type : independent segments : 22000 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 3200 clients per node : 320 repetitions : 1 xfersize : 47008 bytes blocksize : 47008 bytes aggregate filesize : 3.01 TiB stonewallingTime : 300 stoneWallingWearOut : 1 Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- stonewalling pairs accessed min: 14117 max: 22000 -- min data: 0.6 GiB mean data: 0.8 GiB time: 300.8s write 8915 45.91 45.91 0.168726 353.82 0.012549 354.02 0 Max Write: 8914.88 MiB/sec (9347.93 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum write 8914.88 8914.88 8914.88 0.00 198858.33 198858.33 198858.33 0.00 354.02087 0 3200 320 1 0 1 1 0 0 22000 47008 47008 3156054.8 POSIX 0 Finished : Thu May 16 17:50:42 2019 [MDTEST HARD WRITE] time: 1558043443.246664868 -- started at 05/16/2019 17:50:50 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "22000" "-C" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./io500-datadir_10gpu//mdt_hard" "-x" "./io500-datadir_10gpu//mdt_hard-stonewall" "-W" "300" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 65.4% Inodes: 637.8 Mi Used Inodes: 22.6% 3200 tasks, 70400000 files Continue stonewall hit min: 370 max: 385 avg: 374.0 SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 3969.453 3969.425 3969.436 0.004 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 299.059 299.059 299.059 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/16/2019 17:56:00 -- [PFIND EASY] time: 1558043761.029512099 [9] rate: 2.552 kiops time: 70.2s found: 774 (scanned 179097 files, scanned dirents: 165000, unknown dirents: 0) [4] rate: 5.540 kiops time: 70.2s found: 771 (scanned 388759 files, scanned dirents: 390000, unknown dirents: 0) [15] rate: 7.037 kiops time: 70.2s found: 233 (scanned 493859 files, scanned dirents: 495000, unknown dirents: 0) [0] rate: 17.087 kiops time: 70.2s found: 0 (scanned 1199103 files, scanned dirents: 1232012, unknown dirents: 0) [5] rate: 6.199 kiops time: 70.2s found: 425 (scanned 435015 files, scanned dirents: 435000, unknown dirents: 0) [16] rate: 4.584 kiops time: 70.2s found: 201 (scanned 321664 files, scanned dirents: 315000, unknown dirents: 0) [10] rate: 3.573 kiops time: 70.2s found: 844 (scanned 250717 files, scanned dirents: 255000, unknown dirents: 0) [17] rate: 4.133 kiops time: 70.2s found: 36 (scanned 290059 files, scanned dirents: 300000, unknown dirents: 0) [3] rate: 2.414 kiops time: 70.2s found: 1741 (scanned 169401 files, scanned dirents: 156402, unknown dirents: 0) [20] rate: 3.874 kiops time: 70.2s found: 360 (scanned 271890 files, scanned dirents: 270000, unknown dirents: 0) [6] rate: 4.359 kiops time: 70.2s found: 365 (scanned 305898 files, scanned dirents: 315000, unknown dirents: 0) [18] rate: 5.640 kiops time: 70.2s found: 0 (scanned 395761 files, scanned dirents: 405000, unknown dirents: 0) [11] rate: 2.977 kiops time: 70.2s found: 812 (scanned 208893 files, scanned dirents: 195000, unknown dirents: 0) [27] rate: 5.133 kiops time: 70.2s found: 53 (scanned 360192 files, scanned dirents: 360000, unknown dirents: 0) [2] rate: 4.156 kiops time: 70.2s found: 2389 (scanned 291623 files, scanned dirents: 285000, unknown dirents: 0) [12] rate: 3.422 kiops time: 70.2s found: 580 (scanned 240178 files, scanned dirents: 255000, unknown dirents: 0) [23] rate: 5.169 kiops time: 70.2s found: 113 (scanned 362746 files, scanned dirents: 360000, unknown dirents: 0) [1] rate: 5.241 kiops time: 70.2s found: 4480 (scanned 367780 files, scanned dirents: 360000, unknown dirents: 0) [8] rate: 4.573 kiops time: 70.2s found: 151 (scanned 320901 files, scanned dirents: 315000, unknown dirents: 0) [7] rate: 3.550 kiops time: 70.2s found: 257 (scanned 249104 files, scanned dirents: 255000, unknown dirents: 0) [13] rate: 4.392 kiops time: 70.2s found: 288 (scanned 308215 files, scanned dirents: 315000, unknown dirents: 0) [14] rate: 3.758 kiops time: 70.2s found: 159 (scanned 263710 files, scanned dirents: 255000, unknown dirents: 0) [38] rate: 5.769 kiops time: 70.2s found: 19 (scanned 404859 files, scanned dirents: 405000, unknown dirents: 0) [25] rate: 6.371 kiops time: 70.2s found: 4 (scanned 447066 files, scanned dirents: 450000, unknown dirents: 0) [33] rate: 5.093 kiops time: 70.2s found: 20 (scanned 357377 files, scanned dirents: 375000, unknown dirents: 0) [49] rate: 4.789 kiops time: 70.2s found: 12 (scanned 336043 files, scanned dirents: 330000, unknown dirents: 0) [66] rate: 4.117 kiops time: 70.2s found: 24 (scanned 288906 files, scanned dirents: 285000, unknown dirents: 0) [31] rate: 4.334 kiops time: 70.2s found: 34 (scanned 304142 files, scanned dirents: 315000, unknown dirents: 0) [39] rate: 3.556 kiops time: 70.2s found: 37 (scanned 249525 files, scanned dirents: 240000, unknown dirents: 0) [50] rate: 3.110 kiops time: 70.2s found: 44 (scanned 218252 files, scanned dirents: 210000, unknown dirents: 0) [70] rate: 4.518 kiops time: 70.2s found: 0 (scanned 317054 files, scanned dirents: 315000, unknown dirents: 0) [22] rate: 4.109 kiops time: 70.2s found: 39 (scanned 288334 files, scanned dirents: 285000, unknown dirents: 0) [32] rate: 3.776 kiops time: 70.2s found: 117 (scanned 265011 files, scanned dirents: 255000, unknown dirents: 0) [53] rate: 4.080 kiops time: 70.2s found: 308 (scanned 286320 files, scanned dirents: 285000, unknown dirents: 0) [67] rate: 5.207 kiops time: 70.2s found: 8 (scanned 365421 files, scanned dirents: 360000, unknown dirents: 0) [29] rate: 5.309 kiops time: 70.2s found: 125 (scanned 372564 files, scanned dirents: 375000, unknown dirents: 0) [43] rate: 3.495 kiops time: 70.2s found: 344 (scanned 245245 files, scanned dirents: 240000, unknown dirents: 0) [57] rate: 5.010 kiops time: 70.2s found: 3 (scanned 351574 files, scanned dirents: 345000, unknown dirents: 0) [80] rate: 5.109 kiops time: 70.2s found: 43 (scanned 358524 files, scanned dirents: 360000, unknown dirents: 0) [19] rate: 4.946 kiops time: 70.2s found: 80 (scanned 347082 files, scanned dirents: 360000, unknown dirents: 0) [44] rate: 3.688 kiops time: 70.2s found: 8 (scanned 258823 files, scanned dirents: 255000, unknown dirents: 0) [58] rate: 3.492 kiops time: 70.2s found: 76 (scanned 245042 files, scanned dirents: 240000, unknown dirents: 0) [82] rate: 4.318 kiops time: 70.2s found: 9 (scanned 303031 files, scanned dirents: 300000, unknown dirents: 0) [21] rate: 4.762 kiops time: 70.2s found: 128 (scanned 334180 files, scanned dirents: 345000, unknown dirents: 0) [41] rate: 5.099 kiops time: 70.2s found: 21 (scanned 357820 files, scanned dirents: 360000, unknown dirents: 0) [59] rate: 3.751 kiops time: 70.2s found: 102 (scanned 263239 files, scanned dirents: 270000, unknown dirents: 0) [84] rate: 5.473 kiops time: 70.2s found: 10 (scanned 384059 files, scanned dirents: 390000, unknown dirents: 0) [30] rate: 3.024 kiops time: 70.2s found: 228 (scanned 212217 files, scanned dirents: 210000, unknown dirents: 0) [35] rate: 5.334 kiops time: 70.2s found: 132 (scanned 374322 files, scanned dirents: 375000, unknown dirents: 0) [55] rate: 6.057 kiops time: 70.2s found: 197 (scanned 425078 files, scanned dirents: 420000, unknown dirents: 0) [90] rate: 4.090 kiops time: 70.2s found: 115 (scanned 287046 files, scanned dirents: 300000, unknown dirents: 0) [24] rate: 4.536 kiops time: 70.2s found: 412 (scanned 318336 files, scanned dirents: 330000, unknown dirents: 0) [36] rate: 4.177 kiops time: 70.2s found: 14 (scanned 293156 files, scanned dirents: 285000, unknown dirents: 0) [93] rate: 3.100 kiops time: 70.2s found: 236 (scanned 217568 files, scanned dirents: 210000, unknown dirents: 0) [26] rate: 2.607 kiops time: 70.2s found: 81 (scanned 182945 files, scanned dirents: 195000, unknown dirents: 0) [37] rate: 4.142 kiops time: 70.2s found: 7 (scanned 290669 files, scanned dirents: 285000, unknown dirents: 0) [60] rate: 3.717 kiops time: 70.2s found: 180 (scanned 260848 files, scanned dirents: 255000, unknown dirents: 0) [101] rate: 4.925 kiops time: 70.2s found: 74 (scanned 345593 files, scanned dirents: 345000, unknown dirents: 0) [28] rate: 4.319 kiops time: 70.2s found: 121 (scanned 303117 files, scanned dirents: 300000, unknown dirents: 0) [40] rate: 6.159 kiops time: 70.2s found: 4 (scanned 432201 files, scanned dirents: 435000, unknown dirents: 0) [62] rate: 4.332 kiops time: 70.2s found: 26 (scanned 303984 files, scanned dirents: 315000, unknown dirents: 0) [89] rate: 5.726 kiops time: 70.2s found: 79 (scanned 401839 files, scanned dirents: 405000, unknown dirents: 0) [103] rate: 5.776 kiops time: 70.2s found: 20 (scanned 405324 files, scanned dirents: 420000, unknown dirents: 0) [34] rate: 5.565 kiops time: 70.2s found: 3 (scanned 390566 files, scanned dirents: 390000, unknown dirents: 0) [48] rate: 5.739 kiops time: 70.2s found: 447 (scanned 402741 files, scanned dirents: 405000, unknown dirents: 0) [81] rate: 4.464 kiops time: 70.2s found: 8 (scanned 313261 files, scanned dirents: 315000, unknown dirents: 0) [71] rate: 4.318 kiops time: 70.2s found: 8 (scanned 302990 files, scanned dirents: 300000, unknown dirents: 0) [104] rate: 3.238 kiops time: 70.2s found: 25 (scanned 227232 files, scanned dirents: 210000, unknown dirents: 0) [46] rate: 2.912 kiops time: 70.2s found: 494 (scanned 204340 files, scanned dirents: 195000, unknown dirents: 0) [51] rate: 4.937 kiops time: 70.2s found: 4 (scanned 346466 files, scanned dirents: 345000, unknown dirents: 0) [83] rate: 3.728 kiops time: 70.2s found: 309 (scanned 261581 files, scanned dirents: 255000, unknown dirents: 0) [74] rate: 3.259 kiops time: 70.2s found: 12 (scanned 228725 files, scanned dirents: 225000, unknown dirents: 0) [109] rate: 4.313 kiops time: 70.2s found: 86 (scanned 302649 files, scanned dirents: 300000, unknown dirents: 0) [42] rate: 2.479 kiops time: 70.2s found: 319 (scanned 173948 files, scanned dirents: 165000, unknown dirents: 0) [56] rate: 4.334 kiops time: 70.2s found: 12 (scanned 304139 files, scanned dirents: 330000, unknown dirents: 0) [91] rate: 4.572 kiops time: 70.2s found: 85 (scanned 320822 files, scanned dirents: 315000, unknown dirents: 0) [64] rate: 5.184 kiops time: 70.2s found: 4 (scanned 363796 files, scanned dirents: 360000, unknown dirents: 0) [110] rate: 4.175 kiops time: 70.2s found: 62 (scanned 293015 files, scanned dirents: 285000, unknown dirents: 0) [47] rate: 4.093 kiops time: 70.2s found: 19 (scanned 287219 files, scanned dirents: 300000, unknown dirents: 0) [61] rate: 4.570 kiops time: 70.2s found: 16 (scanned 320688 files, scanned dirents: 315000, unknown dirents: 0) [92] rate: 4.529 kiops time: 70.2s found: 83 (scanned 317846 files, scanned dirents: 315000, unknown dirents: 0) [69] rate: 4.132 kiops time: 70.2s found: 111 (scanned 289980 files, scanned dirents: 285000, unknown dirents: 0) [111] rate: 4.905 kiops time: 70.2s found: 63 (scanned 344213 files, scanned dirents: 345000, unknown dirents: 0) [116] rate: 3.452 kiops time: 70.2s found: 38 (scanned 242235 files, scanned dirents: 255000, unknown dirents: 0) [45] rate: 4.550 kiops time: 70.2s found: 36 (scanned 319337 files, scanned dirents: 315000, unknown dirents: 0) [63] rate: 2.692 kiops time: 70.2s found: 76 (scanned 188880 files, scanned dirents: 180000, unknown dirents: 0) [88] rate: 5.097 kiops time: 70.2s found: 86 (scanned 357672 files, scanned dirents: 360000, unknown dirents: 0) [79] rate: 4.488 kiops time: 70.2s found: 71 (scanned 314984 files, scanned dirents: 315000, unknown dirents: 0) [124] rate: 4.091 kiops time: 70.2s found: 189 (scanned 287102 files, scanned dirents: 285000, unknown dirents: 0) [54] rate: 4.844 kiops time: 70.2s found: 50 (scanned 339914 files, scanned dirents: 330000, unknown dirents: 0) [95] rate: 3.774 kiops time: 70.2s found: 66 (scanned 264827 files, scanned dirents: 255000, unknown dirents: 0) [76] rate: 4.931 kiops time: 70.2s found: 69 (scanned 346038 files, scanned dirents: 345000, unknown dirents: 0) [102] rate: 5.962 kiops time: 70.2s found: 36 (scanned 418376 files, scanned dirents: 420000, unknown dirents: 0) [115] rate: 3.153 kiops time: 70.2s found: 86 (scanned 221270 files, scanned dirents: 225000, unknown dirents: 0) [52] rate: 5.252 kiops time: 70.2s found: 85 (scanned 368585 files, scanned dirents: 375000, unknown dirents: 0) [85] rate: 4.486 kiops time: 70.2s found: 23 (scanned 314789 files, scanned dirents: 315000, unknown dirents: 0) [78] rate: 2.700 kiops time: 70.2s found: 5 (scanned 189452 files, scanned dirents: 195000, unknown dirents: 0) [107] rate: 5.299 kiops time: 70.2s found: 64 (scanned 371866 files, scanned dirents: 375000, unknown dirents: 0) [120] rate: 4.907 kiops time: 70.2s found: 52 (scanned 344326 files, scanned dirents: 345000, unknown dirents: 0) [86] rate: 5.266 kiops time: 70.2s found: 3 (scanned 369564 files, scanned dirents: 375000, unknown dirents: 0) [68] rate: 3.058 kiops time: 70.2s found: 40 (scanned 214600 files, scanned dirents: 210000, unknown dirents: 0) [96] rate: 4.311 kiops time: 70.2s found: 29 (scanned 302530 files, scanned dirents: 300000, unknown dirents: 0) [119] rate: 4.018 kiops time: 70.2s found: 249 (scanned 281955 files, scanned dirents: 270000, unknown dirents: 0) [72] rate: 3.895 kiops time: 70.2s found: 26 (scanned 273345 files, scanned dirents: 270000, unknown dirents: 0) [105] rate: 4.133 kiops time: 70.2s found: 35 (scanned 290017 files, scanned dirents: 285000, unknown dirents: 0) [121] rate: 6.196 kiops time: 70.2s found: 157 (scanned 434836 files, scanned dirents: 435000, unknown dirents: 0) [87] rate: 4.266 kiops time: 70.2s found: 4 (scanned 299354 files, scanned dirents: 300000, unknown dirents: 0) [65] rate: 4.119 kiops time: 70.2s found: 15 (scanned 289050 files, scanned dirents: 285000, unknown dirents: 0) [106] rate: 1.599 kiops time: 70.2s found: 1000 (scanned 112243 files, scanned dirents: 90000, unknown dirents: 0) [114] rate: 3.106 kiops time: 70.2s found: 5 (scanned 217942 files, scanned dirents: 225000, unknown dirents: 0) [94] rate: 4.310 kiops time: 70.2s found: 37 (scanned 302485 files, scanned dirents: 315000, unknown dirents: 0) [77] rate: 4.819 kiops time: 70.2s found: 102 (scanned 338204 files, scanned dirents: 345000, unknown dirents: 0) [98] rate: 4.002 kiops time: 70.2s found: 46 (scanned 280834 files, scanned dirents: 285000, unknown dirents: 0) [75] rate: 3.116 kiops time: 70.2s found: 851 (scanned 218645 files, scanned dirents: 225000, unknown dirents: 0) [100] rate: 5.165 kiops time: 70.2s found: 15 (scanned 362478 files, scanned dirents: 360000, unknown dirents: 0) [73] rate: 5.372 kiops time: 70.2s found: 249 (scanned 376987 files, scanned dirents: 375000, unknown dirents: 0) [99] rate: 4.069 kiops time: 70.2s found: 41 (scanned 285537 files, scanned dirents: 285000, unknown dirents: 0) [122] rate: 2.650 kiops time: 70.2s found: 28 (scanned 185988 files, scanned dirents: 180000, unknown dirents: 0) [137] rate: 2.958 kiops time: 70.2s found: 451 (scanned 207608 files, scanned dirents: 195000, unknown dirents: 0) [108] rate: 4.748 kiops time: 70.2s found: 133 (scanned 333162 files, scanned dirents: 330000, unknown dirents: 0) [123] rate: 3.909 kiops time: 70.2s found: 105 (scanned 274299 files, scanned dirents: 270000, unknown dirents: 0) [140] rate: 2.775 kiops time: 70.2s found: 512 (scanned 194709 files, scanned dirents: 180000, unknown dirents: 0) [97] rate: 5.288 kiops time: 70.2s found: 268 (scanned 371084 files, scanned dirents: 375000, unknown dirents: 0) [125] rate: 3.732 kiops time: 70.2s found: 58 (scanned 261889 files, scanned dirents: 255000, unknown dirents: 0) [130] rate: 3.974 kiops time: 70.2s found: 44 (scanned 278892 files, scanned dirents: 285000, unknown dirents: 0) [126] rate: 2.902 kiops time: 70.2s found: 6 (scanned 203682 files, scanned dirents: 210000, unknown dirents: 0) [138] rate: 5.319 kiops time: 70.2s found: 42 (scanned 373275 files, scanned dirents: 375000, unknown dirents: 0) [127] rate: 2.733 kiops time: 70.2s found: 44 (scanned 191797 files, scanned dirents: 195000, unknown dirents: 0) [141] rate: 3.741 kiops time: 70.2s found: 111 (scanned 262548 files, scanned dirents: 255000, unknown dirents: 0) [113] rate: 3.344 kiops time: 70.2s found: 249 (scanned 234676 files, scanned dirents: 225000, unknown dirents: 0) [159] rate: 2.583 kiops time: 70.2s found: 3626 (scanned 181235 files, scanned dirents: 165000, unknown dirents: 0) [117] rate: 5.131 kiops time: 70.2s found: 3 (scanned 360077 files, scanned dirents: 360000, unknown dirents: 0) [142] rate: 3.396 kiops time: 70.2s found: 266 (scanned 238307 files, scanned dirents: 255000, unknown dirents: 0) [146] rate: 5.989 kiops time: 70.2s found: 50 (scanned 420309 files, scanned dirents: 420000, unknown dirents: 0) [118] rate: 5.755 kiops time: 70.2s found: 0 (scanned 403862 files, scanned dirents: 405000, unknown dirents: 0) [158] rate: 5.516 kiops time: 70.2s found: 375 (scanned 387073 files, scanned dirents: 390000, unknown dirents: 0) [112] rate: 5.021 kiops time: 70.2s found: 54 (scanned 352352 files, scanned dirents: 375000, unknown dirents: 0) [143] rate: 3.722 kiops time: 70.2s found: 29 (scanned 261178 files, scanned dirents: 255000, unknown dirents: 0) [148] rate: 4.545 kiops time: 70.2s found: 61 (scanned 318980 files, scanned dirents: 315000, unknown dirents: 0) [139] rate: 4.133 kiops time: 70.2s found: 50 (scanned 290042 files, scanned dirents: 315000, unknown dirents: 0) [128] rate: 4.341 kiops time: 70.2s found: 49 (scanned 304662 files, scanned dirents: 300000, unknown dirents: 0) [155] rate: 4.274 kiops time: 70.2s found: 81 (scanned 299927 files, scanned dirents: 300000, unknown dirents: 0) [133] rate: 3.089 kiops time: 70.2s found: 94 (scanned 216775 files, scanned dirents: 210000, unknown dirents: 0) [147] rate: 3.882 kiops time: 70.2s found: 50 (scanned 272423 files, scanned dirents: 270000, unknown dirents: 0) [134] rate: 4.309 kiops time: 70.2s found: 14 (scanned 302410 files, scanned dirents: 300000, unknown dirents: 0) [149] rate: 5.718 kiops time: 70.2s found: 72 (scanned 401245 files, scanned dirents: 405000, unknown dirents: 0) [132] rate: 4.321 kiops time: 70.2s found: 471 (scanned 303242 files, scanned dirents: 300000, unknown dirents: 0) [150] rate: 3.968 kiops time: 70.2s found: 27 (scanned 278492 files, scanned dirents: 270000, unknown dirents: 0) [131] rate: 3.282 kiops time: 70.2s found: 36 (scanned 230313 files, scanned dirents: 225000, unknown dirents: 0) [153] rate: 4.023 kiops time: 70.2s found: 72 (scanned 282345 files, scanned dirents: 285000, unknown dirents: 0) [129] rate: 3.978 kiops time: 70.2s found: 60 (scanned 279169 files, scanned dirents: 285000, unknown dirents: 0) [154] rate: 3.266 kiops time: 70.2s found: 20 (scanned 229168 files, scanned dirents: 240000, unknown dirents: 0) [135] rate: 3.704 kiops time: 70.2s found: 63 (scanned 259951 files, scanned dirents: 255000, unknown dirents: 0) [157] rate: 4.183 kiops time: 70.2s found: 1 (scanned 293578 files, scanned dirents: 315000, unknown dirents: 0) [136] rate: 6.368 kiops time: 70.2s found: 43 (scanned 446881 files, scanned dirents: 465000, unknown dirents: 0) [156] rate: 4.122 kiops time: 70.2s found: 31 (scanned 289269 files, scanned dirents: 300000, unknown dirents: 0) [152] rate: 4.394 kiops time: 70.2s found: 288 (scanned 308364 files, scanned dirents: 300000, unknown dirents: 0) [145] rate: 4.716 kiops time: 70.2s found: 122 (scanned 330971 files, scanned dirents: 330000, unknown dirents: 0) [144] rate: 4.674 kiops time: 70.2s found: 51 (scanned 328003 files, scanned dirents: 330000, unknown dirents: 0) [151] rate: 3.152 kiops time: 70.2s found: 18 (scanned 221214 files, scanned dirents: 210000, unknown dirents: 0) [DONE] rate: 702.225 kiops time: 70.2s found: 32907 (scanned 49280203 files, scanned dirents: 49283414, unknown dirents: 0) MATCHED 32907/49280203 time: 1558043832.316813077 [IOR EASY READ] time: 1558043832.318048999 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 16 17:57:22 2019 Command line : ./bin//ior --posix.odirect -t 1m -b 6000m -F -r -R -C -Q 1 -g -G 27 -k -e -o ./io500-datadir_10gpu//ior_easy/ior_file_easy -O stoneWallingStatusFile=./io500-datadir_10gpu//ior_easy/stonewall Machine : Linux f16u15.int.janelia.org TestID : 0 StartTime : Thu May 16 17:57:22 2019 Path : /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu/ior_easy FS : 100.0 TiB Used FS: 65.4% Inodes: 637.8 Mi Used Inodes: 22.8% Options: api : POSIX apiVersion : test filename : ./io500-datadir_10gpu//ior_easy/ior_file_easy access : file-per-process type : independent segments : 1 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 3200 clients per node : 320 repetitions : 1 xfersize : 1 MiB blocksize : 5.86 GiB aggregate filesize : 18.31 TiB Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- read 78923 6144000 1024.00 1.18 242.12 0.037371 243.27 0 Max Read: 78923.14 MiB/sec (82756.91 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum read 78923.14 78923.14 78923.14 0.00 78923.14 78923.14 78923.14 0.00 243.27465 0 3200 320 1 1 1 1 0 0 1 6291456000 1048576 19200000.0 POSIX 0 Finished : Thu May 16 18:01:27 2019 [MDTEST EASY STAT] time: 1558044088.136025107 -- started at 05/16/2019 18:01:35 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "15000" "-u" "-L" "-T" "-F" "-d" "./io500-datadir_10gpu//mdt_easy" "-x" "./io500-datadir_10gpu//mdt_easy-stonewall" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 65.4% Inodes: 637.8 Mi Used Inodes: 22.8% 3200 tasks, 48000000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 2652606.538 2651215.532 2652342.603 190.667 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/16/2019 18:01:53 -- [IOR HARD READ] time: 1558044113.565118820 IOR-3.2.0: MPI Coordinated Test of Parallel I/O Began : Thu May 16 18:02:09 2019 Command line : ./bin//ior --posix.odirect -s 22000 -r -R -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -s 22000 --posix.odirect -o ./io500-datadir_10gpu//ior_hard/IOR_file -O stoneWallingStatusFile=./io500-datadir_10gpu//ior_hard/stonewall Machine : Linux f16u15.int.janelia.org TestID : 0 StartTime : Thu May 16 18:02:09 2019 Path : /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu/ior_hard FS : 100.0 TiB Used FS: 65.4% Inodes: 637.8 Mi Used Inodes: 22.8% Options: api : POSIX apiVersion : test filename : ./io500-datadir_10gpu//ior_hard/IOR_file access : single-shared-file type : independent segments : 22000 ordering in a file : sequential ordering inter file : constant task offset task offset : 1 tasks : 3200 clients per node : 320 repetitions : 1 xfersize : 47008 bytes blocksize : 47008 bytes aggregate filesize : 3.01 TiB Results: access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- read 21252 45.91 45.91 1.14 147.37 0.025044 148.51 0 Max Read: 21252.15 MiB/sec (22284.50 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum read 21252.15 21252.15 21252.15 0.00 474057.58 474057.58 474057.58 0.00 148.50517 0 3200 320 1 0 1 1 0 0 22000 47008 47008 3156054.8 POSIX 0 Finished : Thu May 16 18:04:39 2019 [MDTEST HARD STAT] time: 1558044279.823619259 -- started at 05/16/2019 18:04:49 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "22000" "-T" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./io500-datadir_10gpu//mdt_hard" "-x" "./io500-datadir_10gpu//mdt_hard-stonewall" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 65.4% Inodes: 637.8 Mi Used Inodes: 22.8% 3200 tasks, 70400000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 1953908.465 1935450.746 1942583.795 1432.603 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/16/2019 18:04:50 -- [MDTEST EASY DELETE] time: 1558044290.821092565 -- started at 05/16/2019 18:05:11 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "15000" "-u" "-L" "-r" "-F" "-d" "./io500-datadir_10gpu//mdt_easy" "-x" "./io500-datadir_10gpu//mdt_easy-stonewall" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 65.4% Inodes: 637.8 Mi Used Inodes: 22.8% 3200 tasks, 48000000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 114503.294 114500.115 114502.291 0.188 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.366 0.366 0.366 0.000 -- finished at 05/16/2019 18:12:25 -- [MDTEST HARD READ] time: 1558044745.631036165 -- started at 05/16/2019 18:12:35 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "22000" "-E" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./io500-datadir_10gpu//mdt_hard" "-x" "./io500-datadir_10gpu//mdt_hard-stonewall" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 65.2% Inodes: 637.8 Mi Used Inodes: 14.7% 3200 tasks, 70400000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 634137.295 632287.071 632651.391 121.167 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 -- finished at 05/16/2019 18:12:37 -- [MDTEST HARD DELETE] time: 1558044757.927716839 -- started at 05/16/2019 18:12:47 -- mdtest-1.9.3 was launched with 3200 total task(s) on 10 node(s) Command line used: ./bin//mdtest "--posix.odirect" "-n" "22000" "-r" "-t" "-F" "-w" "3901" "-e" "3901" "-d" "./io500-datadir_10gpu//mdt_hard" "-x" "./io500-datadir_10gpu//mdt_hard-stonewall" Path: /misc/weka/wekaFS_1/io-500-script/io500-datadir_10gpu FS: 100.0 TiB Used FS: 65.2% Inodes: 637.8 Mi Used Inodes: 14.7% 3200 tasks, 70400000 files SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 0.000 0.000 0.000 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 3706.570 3706.029 3706.552 0.025 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.263 0.263 0.263 0.000 -- finished at 05/16/2019 18:18:23 -- Deleting IO-500 ./io500-datadir_10gpu/ time: 1558045104.415773464 [END TIME] 2019-05-16 18:19:51-04:00 [IO-500 COMPLETED] Result also in config-gpu_10-result-1558042164.txt; Now use io-500-score.sh to compute the score