- ior_easy_read
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Mon Jul 6 21:25:14 2020
Command line : /opt/software/rsc/io500-2020-spring/io500-app/bin/ior -r -R -a POSIX -t 2m --posix.odirect -v -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy/ior_file_easy -O stoneWallingStatusFile=/lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy/stonewall
Machine : Linux n02p001.gvr.local
Start time skew across all tasks: 132605.33 sec
TestID : 0
StartTime : Mon Jul 6 21:25:14 2020
Path : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy
FS : 259.3 TiB Used FS: 28.0% Inodes: 1204.8 Mi Used Inodes: 16.0%
Participating tasks: 800
Using reorderTasks '-C' (useful to avoid read cache in client)
Options:
api : POSIX
apiVersion :
test filename : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy/ior_file_easy
access : file-per-process
type : independent
segments : 1
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 50
tasks : 800
clients per node : 16
repetitions : 1
xfersize : 2 MiB
blocksize : 9.46 TiB
aggregate filesize : 7568.36 TiB
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
Commencing read performance test: Mon Jul 6 21:25:16 2020
WARNING: Expected aggregate file size = 8321499136000000.
WARNING: Stat() of aggregate file size = 75160249958400.
WARNING: Using actual aggregate bytes moved = 75160249958400.
read 80674 40378 0.000179 10158080000 2048.00 0.521955 887.59 0.554425 888.50 0
Max Read: 80673.70 MiB/sec (84592.51 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
read 80673.70 80673.70 80673.70 0.00 40336.85 40336.85 40336.85 0.00 888.49770 NA NA 0 800 16 1 1 1 1 0 0 1 10401873920000 2097152 71678400.0 POSIX 0
Finished : Mon Jul 6 21:40:06 2020
- ior_easy_write
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Mon Jul 6 20:43:31 2020
Command line : /opt/software/rsc/io500-2020-spring/io500-app/bin/ior -w -a POSIX -t 2m --posix.odirect -v -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy/ior_file_easy -O stoneWallingStatusFile=/lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300
Machine : Linux n02p001.gvr.local
Start time skew across all tasks: 132605.05 sec
TestID : 0
StartTime : Mon Jul 6 20:43:31 2020
Path : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy
FS : 259.3 TiB Used FS: 0.1% Inodes: 1047.8 Mi Used Inodes: 0.0%
Participating tasks: 800
Using reorderTasks '-C' (useful to avoid read cache in client)
Options:
api : POSIX
apiVersion :
test filename : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_easy/ior_file_easy
access : file-per-process
type : independent
segments : 1
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 50
tasks : 800
clients per node : 16
repetitions : 1
xfersize : 2 MiB
blocksize : 9.46 TiB
aggregate filesize : 7568.36 TiB
stonewallingTime : 300
stoneWallingWearOut : 1
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
Commencing write performance test: Mon Jul 6 20:43:33 2020
400: stonewalling pairs accessed: 30458
256: stonewalling pairs accessed: 31721
272: stonewalling pairs accessed: 36343
97: stonewalling pairs accessed: 41483
405: stonewalling pairs accessed: 36357
118: stonewalling pairs accessed: 34599
413: stonewalling pairs accessed: 36218
314: stonewalling pairs accessed: 30336
179: stonewalling pairs accessed: 36945
745: stonewalling pairs accessed: 32432
271: stonewalling pairs accessed: 36546
276: stonewalling pairs accessed: 30700
403: stonewalling pairs accessed: 31441
401: stonewalling pairs accessed: 30292
261: stonewalling pairs accessed: 30993
119: stonewalling pairs accessed: 36321
410: stonewalling pairs accessed: 31232
304: stonewalling pairs accessed: 31122
187: stonewalling pairs accessed: 36511
274: stonewalling pairs accessed: 37031
181: stonewalling pairs accessed: 36948
736: stonewalling pairs accessed: 32002
414: stonewalling pairs accessed: 33092
319: stonewalling pairs accessed: 31431
599: stonewalling pairs accessed: 33503
311: stonewalling pairs accessed: 33123
746: stonewalling pairs accessed: 37791
214: stonewalling pairs accessed: 30343
288: stonewalling pairs accessed: 36343
407: stonewalling pairs accessed: 31620
318: stonewalling pairs accessed: 31886
279: stonewalling pairs accessed: 33201
386: stonewalling pairs accessed: 30509
176: stonewalling pairs accessed: 32948
302: stonewalling pairs accessed: 31213
23: stonewalling pairs accessed: 39799
404: stonewalling pairs accessed: 29828
306: stonewalling pairs accessed: 30957
398: stonewalling pairs accessed: 30256
751: stonewalling pairs accessed: 30259
262: stonewalling pairs accessed: 36604
147: stonewalling pairs accessed: 34305
411: stonewalling pairs accessed: 30180
175: stonewalling pairs accessed: 34655
740: stonewalling pairs accessed: 32534
121: stonewalling pairs accessed: 33168
188: stonewalling pairs accessed: 32011
259: stonewalling pairs accessed: 35998
96: stonewalling pairs accessed: 33007
735: stonewalling pairs accessed: 32853
268: stonewalling pairs accessed: 30770
222: stonewalling pairs accessed: 31561
737: stonewalling pairs accessed: 32258
127: stonewalling pairs accessed: 33849
312: stonewalling pairs accessed: 32260
396: stonewalling pairs accessed: 31055
741: stonewalling pairs accessed: 32034
309: stonewalling pairs accessed: 32075
286: stonewalling pairs accessed: 32415
189: stonewalling pairs accessed: 31421
48: stonewalling pairs accessed: 36946
695: stonewalling pairs accessed: 32789
387: stonewalling pairs accessed: 29884
105: stonewalling pairs accessed: 32189
681: stonewalling pairs accessed: 34852
296: stonewalling pairs accessed: 33068
742: stonewalling pairs accessed: 32399
113: stonewalling pairs accessed: 30910
315: stonewalling pairs accessed: 32545
394: stonewalling pairs accessed: 29917
108: stonewalling pairs accessed: 41391
269: stonewalling pairs accessed: 32046
208: stonewalling pairs accessed: 30373
25: stonewalling pairs accessed: 30440
408: stonewalling pairs accessed: 35934
739: stonewalling pairs accessed: 32455
671: stonewalling pairs accessed: 40105
125: stonewalling pairs accessed: 32424
283: stonewalling pairs accessed: 31153
258: stonewalling pairs accessed: 32808
399: stonewalling pairs accessed: 31638
99: stonewalling pairs accessed: 31862
185: stonewalling pairs accessed: 31679
126: stonewalling pairs accessed: 34517
384: stonewalling pairs accessed: 30773
100: stonewalling pairs accessed: 41870
260: stonewalling pairs accessed: 31075
16: stonewalling pairs accessed: 34322
284: stonewalling pairs accessed: 36795
190: stonewalling pairs accessed: 31170
738: stonewalling pairs accessed: 32615
168: stonewalling pairs accessed: 32249
112: stonewalling pairs accessed: 31957
257: stonewalling pairs accessed: 30850
107: stonewalling pairs accessed: 41588
393: stonewalling pairs accessed: 30502
273: stonewalling pairs accessed: 36748
216: stonewalling pairs accessed: 35033
295: stonewalling pairs accessed: 33620
664: stonewalling pairs accessed: 40107
747: stonewalling pairs accessed: 31557
186: stonewalling pairs accessed: 32875
656: stonewalling pairs accessed: 31617
117: stonewalling pairs accessed: 33891
630: stonewalling pairs accessed: 39173
391: stonewalling pairs accessed: 30437
270: stonewalling pairs accessed: 31623
691: stonewalling pairs accessed: 33777
104: stonewalling pairs accessed: 32475
607: stonewalling pairs accessed: 32351
182: stonewalling pairs accessed: 31868
122: stonewalling pairs accessed: 33309
80: stonewalling pairs accessed: 34200
715: stonewalling pairs accessed: 30023
267: stonewalling pairs accessed: 32341
694: stonewalling pairs accessed: 33473
667: stonewalling pairs accessed: 35299
392: stonewalling pairs accessed: 31126
606: stonewalling pairs accessed: 33521
110: stonewalling pairs accessed: 33821
191: stonewalling pairs accessed: 30780
796: stonewalling pairs accessed: 36419
619: stonewalling pairs accessed: 30412
124: stonewalling pairs accessed: 34353
415: stonewalling pairs accessed: 35700
128: stonewalling pairs accessed: 39557
266: stonewalling pairs accessed: 31818
653: stonewalling pairs accessed: 38872
697: stonewalling pairs accessed: 32486
666: stonewalling pairs accessed: 32647
592: stonewalling pairs accessed: 32513
102: stonewalling pairs accessed: 35603
173: stonewalling pairs accessed: 34619
385: stonewalling pairs accessed: 32158
20: stonewalling pairs accessed: 31890
303: stonewalling pairs accessed: 36013
277: stonewalling pairs accessed: 30512
123: stonewalling pairs accessed: 35381
402: stonewalling pairs accessed: 35977
631: stonewalling pairs accessed: 32512
342: stonewalling pairs accessed: 30288
689: stonewalling pairs accessed: 31775
265: stonewalling pairs accessed: 32348
217: stonewalling pairs accessed: 31332
109: stonewalling pairs accessed: 32603
180: stonewalling pairs accessed: 31709
305: stonewalling pairs accessed: 36189
598: stonewalling pairs accessed: 32177
300: stonewalling pairs accessed: 31723
285: stonewalling pairs accessed: 31365
720: stonewalling pairs accessed: 33807
346: stonewalling pairs accessed: 30203
687: stonewalling pairs accessed: 33355
114: stonewalling pairs accessed: 32901
406: stonewalling pairs accessed: 36388
641: stonewalling pairs accessed: 39600
169: stonewalling pairs accessed: 30100
60: stonewalling pairs accessed: 41283
628: stonewalling pairs accessed: 34969
377: stonewalling pairs accessed: 38687
30: stonewalling pairs accessed: 33042
299: stonewalling pairs accessed: 30518
704: stonewalling pairs accessed: 31382
604: stonewalling pairs accessed: 36062
62: stonewalling pairs accessed: 41215
784: stonewalling pairs accessed: 29912
752: stonewalling pairs accessed: 41470
412: stonewalling pairs accessed: 36035
310: stonewalling pairs accessed: 35896
86: stonewalling pairs accessed: 35629
281: stonewalling pairs accessed: 35939
698: stonewalling pairs accessed: 34039
101: stonewalling pairs accessed: 40782
178: stonewalling pairs accessed: 37084
347: stonewalling pairs accessed: 31638
673: stonewalling pairs accessed: 30851
639: stonewalling pairs accessed: 31505
594: stonewalling pairs accessed: 32468
293: stonewalling pairs accessed: 31647
160: stonewalling pairs accessed: 31232
280: stonewalling pairs accessed: 31307
688: stonewalling pairs accessed: 31998
221: stonewalling pairs accessed: 35388
657: stonewalling pairs accessed: 32329
98: stonewalling pairs accessed: 41605
183: stonewalling pairs accessed: 36810
21: stonewalling pairs accessed: 34112
643: stonewalling pairs accessed: 40112
301: stonewalling pairs accessed: 31265
219: stonewalling pairs accessed: 30403
601: stonewalling pairs accessed: 32422
282: stonewalling pairs accessed: 30993
699: stonewalling pairs accessed: 31987
248: stonewalling pairs accessed: 31123
654: stonewalling pairs accessed: 34029
106: stonewalling pairs accessed: 40274
670: stonewalling pairs accessed: 33877
748: stonewalling pairs accessed: 37530
797: stonewalling pairs accessed: 38114
31: stonewalling pairs accessed: 34143
732: stonewalling pairs accessed: 34108
596: stonewalling pairs accessed: 35801
409: stonewalling pairs accessed: 36218
223: stonewalling pairs accessed: 30692
55: stonewalling pairs accessed: 32467
337: stonewalling pairs accessed: 32115
111: stonewalling pairs accessed: 41255
313: stonewalling pairs accessed: 36045
289: stonewalling pairs accessed: 31438
18: stonewalling pairs accessed: 32932
661: stonewalling pairs accessed: 33870
120: stonewalling pairs accessed: 38378
278: stonewalling pairs accessed: 30984
635: stonewalling pairs accessed: 31456
136: stonewalling pairs accessed: 34201
95: stonewalling pairs accessed: 33894
597: stonewalling pairs accessed: 32854
171: stonewalling pairs accessed: 30583
502: stonewalling pairs accessed: 35034
220: stonewalling pairs accessed: 32232
197: stonewalling pairs accessed: 31426
703: stonewalling pairs accessed: 32952
199: stonewalling pairs accessed: 29748
103: stonewalling pairs accessed: 32243
308: stonewalling pairs accessed: 35646
364: stonewalling pairs accessed: 32038
24: stonewalling pairs accessed: 34377
440: stonewalling pairs accessed: 31017
287: stonewalling pairs accessed: 36523
603: stonewalling pairs accessed: 32960
637: stonewalling pairs accessed: 31950
789: stonewalling pairs accessed: 31228
213: stonewalling pairs accessed: 32387
702: stonewalling pairs accessed: 31930
139: stonewalling pairs accessed: 34319
84: stonewalling pairs accessed: 34177
184: stonewalling pairs accessed: 36723
588: stonewalling pairs accessed: 40456
662: stonewalling pairs accessed: 31771
170: stonewalling pairs accessed: 31140
538: stonewalling pairs accessed: 33809
434: stonewalling pairs accessed: 32013
634: stonewalling pairs accessed: 31251
593: stonewalling pairs accessed: 31184
692: stonewalling pairs accessed: 38508
716: stonewalling pairs accessed: 33879
371: stonewalling pairs accessed: 38388
177: stonewalling pairs accessed: 36930
675: stonewalling pairs accessed: 33261
650: stonewalling pairs accessed: 38603
669: stonewalling pairs accessed: 33172
750: stonewalling pairs accessed: 38454
464: stonewalling pairs accessed: 31131
504: stonewalling pairs accessed: 32213
316: stonewalling pairs accessed: 36206
44: stonewalling pairs accessed: 41974
2: stonewalling pairs accessed: 30187
192: stonewalling pairs accessed: 32256
163: stonewalling pairs accessed: 31521
730: stonewalling pairs accessed: 32480
766: stonewalling pairs accessed: 40943
595: stonewalling pairs accessed: 42256
94: stonewalling pairs accessed: 33383
705: stonewalling pairs accessed: 33975
215: stonewalling pairs accessed: 32011
252: stonewalling pairs accessed: 36133
58: stonewalling pairs accessed: 40537
372: stonewalling pairs accessed: 31392
141: stonewalling pairs accessed: 32915
28: stonewalling pairs accessed: 34769
116: stonewalling pairs accessed: 39872
166: stonewalling pairs accessed: 31443
686: stonewalling pairs accessed: 31287
83: stonewalling pairs accessed: 32011
733: stonewalling pairs accessed: 31276
602: stonewalling pairs accessed: 44409
0: stonewalling pairs accessed: 34857
509: stonewalling pairs accessed: 33814
26: stonewalling pairs accessed: 33735
35: stonewalling pairs accessed: 36790
792: stonewalling pairs accessed: 35121
710: stonewalling pairs accessed: 30264
754: stonewalling pairs accessed: 38787
57: stonewalling pairs accessed: 30136
307: stonewalling pairs accessed: 36146
292: stonewalling pairs accessed: 35934
174: stonewalling pairs accessed: 31786
679: stonewalling pairs accessed: 34843
90: stonewalling pairs accessed: 35697
438: stonewalling pairs accessed: 32083
37: stonewalling pairs accessed: 32428
77: stonewalling pairs accessed: 41423
731: stonewalling pairs accessed: 33476
275: stonewalling pairs accessed: 36766
711: stonewalling pairs accessed: 32691
19: stonewalling pairs accessed: 40312
793: stonewalling pairs accessed: 32159
665: stonewalling pairs accessed: 34120
243: stonewalling pairs accessed: 36421
263: stonewalling pairs accessed: 36244
465: stonewalling pairs accessed: 31648
496: stonewalling pairs accessed: 30142
640: stonewalling pairs accessed: 40453
50: stonewalling pairs accessed: 36553
200: stonewalling pairs accessed: 33106
291: stonewalling pairs accessed: 36192
582: stonewalling pairs accessed: 42972
161: stonewalling pairs accessed: 32252
87: stonewalling pairs accessed: 32258
46: stonewalling pairs accessed: 31771
444: stonewalling pairs accessed: 31234
672: stonewalling pairs accessed: 33018
734: stonewalling pairs accessed: 38145
758: stonewalling pairs accessed: 31373
642: stonewalling pairs accessed: 33647
636: stonewalling pairs accessed: 30916
600: stonewalling pairs accessed: 44799
788: stonewalling pairs accessed: 33235
67: stonewalling pairs accessed: 38000
56: stonewalling pairs accessed: 32409
712: stonewalling pairs accessed: 31541
690: stonewalling pairs accessed: 39236
88: stonewalling pairs accessed: 32916
204: stonewalling pairs accessed: 30813
724: stonewalling pairs accessed: 31233
447: stonewalling pairs accessed: 31736
317: stonewalling pairs accessed: 35996
298: stonewalling pairs accessed: 35386
798: stonewalling pairs accessed: 32197
684: stonewalling pairs accessed: 32272
589: stonewalling pairs accessed: 42073
249: stonewalling pairs accessed: 30543
242: stonewalling pairs accessed: 31220
472: stonewalling pairs accessed: 31862
327: stonewalling pairs accessed: 32535
164: stonewalling pairs accessed: 31518
706: stonewalling pairs accessed: 31075
69: stonewalling pairs accessed: 30327
749: stonewalling pairs accessed: 38551
42: stonewalling pairs accessed: 34651
22: stonewalling pairs accessed: 39756
576: stonewalling pairs accessed: 34737
205: stonewalling pairs accessed: 30441
608: stonewalling pairs accessed: 32572
682: stonewalling pairs accessed: 32642
433: stonewalling pairs accessed: 31681
799: stonewalling pairs accessed: 32538
210: stonewalling pairs accessed: 35789
53: stonewalling pairs accessed: 32036
470: stonewalling pairs accessed: 33412
167: stonewalling pairs accessed: 31403
43: stonewalling pairs accessed: 31251
725: stonewalling pairs accessed: 32541
143: stonewalling pairs accessed: 32279
708: stonewalling pairs accessed: 32337
583: stonewalling pairs accessed: 32705
651: stonewalling pairs accessed: 39381
294: stonewalling pairs accessed: 36078
663: stonewalling pairs accessed: 38929
627: stonewalling pairs accessed: 39466
795: stonewalling pairs accessed: 31835
685: stonewalling pairs accessed: 31964
500: stonewalling pairs accessed: 34781
475: stonewalling pairs accessed: 31906
759: stonewalling pairs accessed: 32938
165: stonewalling pairs accessed: 31404
196: stonewalling pairs accessed: 31085
138: stonewalling pairs accessed: 32578
85: stonewalling pairs accessed: 40176
255: stonewalling pairs accessed: 31389
36: stonewalling pairs accessed: 32901
422: stonewalling pairs accessed: 30723
649: stonewalling pairs accessed: 34176
728: stonewalling pairs accessed: 32739
613: stonewalling pairs accessed: 34460
54: stonewalling pairs accessed: 35772
446: stonewalling pairs accessed: 32172
498: stonewalling pairs accessed: 30286
794: stonewalling pairs accessed: 36980
129: stonewalling pairs accessed: 33998
585: stonewalling pairs accessed: 32286
201: stonewalling pairs accessed: 31552
45: stonewalling pairs accessed: 39572
244: stonewalling pairs accessed: 30774
389: stonewalling pairs accessed: 35129
290: stonewalling pairs accessed: 36113
569: stonewalling pairs accessed: 41724
674: stonewalling pairs accessed: 31417
59: stonewalling pairs accessed: 40468
14: stonewalling pairs accessed: 31778
648: stonewalling pairs accessed: 30136
727: stonewalling pairs accessed: 31612
659: stonewalling pairs accessed: 37354
75: stonewalling pairs accessed: 34958
499: stonewalling pairs accessed: 33804
202: stonewalling pairs accessed: 30703
172: stonewalling pairs accessed: 34797
41: stonewalling pairs accessed: 32549
586: stonewalling pairs accessed: 32965
676: stonewalling pairs accessed: 32000
356: stonewalling pairs accessed: 31023
757: stonewalling pairs accessed: 32558
241: stonewalling pairs accessed: 30127
140: stonewalling pairs accessed: 33788
209: stonewalling pairs accessed: 35688
787: stonewalling pairs accessed: 35946
621: stonewalling pairs accessed: 30966
321: stonewalling pairs accessed: 31868
383: stonewalling pairs accessed: 30728
701: stonewalling pairs accessed: 33470
71: stonewalling pairs accessed: 35519
63: stonewalling pairs accessed: 40263
503: stonewalling pairs accessed: 32565
461: stonewalling pairs accessed: 31132
441: stonewalling pairs accessed: 32523
683: stonewalling pairs accessed: 32107
194: stonewalling pairs accessed: 30852
39: stonewalling pairs accessed: 35020
3: stonewalling pairs accessed: 31570
644: stonewalling pairs accessed: 30246
761: stonewalling pairs accessed: 34619
254: stonewalling pairs accessed: 32890
615: stonewalling pairs accessed: 32309
722: stonewalling pairs accessed: 40886
328: stonewalling pairs accessed: 31488
528: stonewalling pairs accessed: 41781
591: stonewalling pairs accessed: 40387
142: stonewalling pairs accessed: 34735
195: stonewalling pairs accessed: 32711
497: stonewalling pairs accessed: 33404
514: stonewalling pairs accessed: 32843
64: stonewalling pairs accessed: 35865
760: stonewalling pairs accessed: 33016
370: stonewalling pairs accessed: 32732
626: stonewalling pairs accessed: 40008
652: stonewalling pairs accessed: 32220
38: stonewalling pairs accessed: 40418
218: stonewalling pairs accessed: 36079
7: stonewalling pairs accessed: 32633
680: stonewalling pairs accessed: 38201
623: stonewalling pairs accessed: 34114
331: stonewalling pairs accessed: 30925
575: stonewalling pairs accessed: 33676
245: stonewalling pairs accessed: 30146
264: stonewalling pairs accessed: 36678
721: stonewalling pairs accessed: 33194
581: stonewalling pairs accessed: 33086
696: stonewalling pairs accessed: 40867
505: stonewalling pairs accessed: 31654
198: stonewalling pairs accessed: 30493
162: stonewalling pairs accessed: 34295
753: stonewalling pairs accessed: 30035
4: stonewalling pairs accessed: 32379
655: stonewalling pairs accessed: 32598
638: stonewalling pairs accessed: 39562
70: stonewalling pairs accessed: 42365
478: stonewalling pairs accessed: 31809
517: stonewalling pairs accessed: 34183
376: stonewalling pairs accessed: 31033
622: stonewalling pairs accessed: 31786
579: stonewalling pairs accessed: 35771
564: stonewalling pairs accessed: 32314
501: stonewalling pairs accessed: 32532
133: stonewalling pairs accessed: 39824
206: stonewalling pairs accessed: 37261
790: stonewalling pairs accessed: 32622
435: stonewalling pairs accessed: 32086
765: stonewalling pairs accessed: 34509
743: stonewalling pairs accessed: 38924
324: stonewalling pairs accessed: 31351
647: stonewalling pairs accessed: 31810
212: stonewalling pairs accessed: 36304
367: stonewalling pairs accessed: 31048
11: stonewalling pairs accessed: 32021
612: stonewalling pairs accessed: 37930
476: stonewalling pairs accessed: 33481
51: stonewalling pairs accessed: 40713
72: stonewalling pairs accessed: 41344
375: stonewalling pairs accessed: 31968
27: stonewalling pairs accessed: 40255
203: stonewalling pairs accessed: 37497
395: stonewalling pairs accessed: 34737
755: stonewalling pairs accessed: 33623
330: stonewalling pairs accessed: 30970
578: stonewalling pairs accessed: 37299
781: stonewalling pairs accessed: 37718
508: stonewalling pairs accessed: 33345
467: stonewalling pairs accessed: 30688
560: stonewalling pairs accessed: 34205
718: stonewalling pairs accessed: 37459
611: stonewalling pairs accessed: 30251
247: stonewalling pairs accessed: 36305
6: stonewalling pairs accessed: 33377
78: stonewalling pairs accessed: 40698
61: stonewalling pairs accessed: 40148
590: stonewalling pairs accessed: 32734
207: stonewalling pairs accessed: 37305
388: stonewalling pairs accessed: 34345
529: stonewalling pairs accessed: 41710
382: stonewalling pairs accessed: 30395
629: stonewalling pairs accessed: 39652
510: stonewalling pairs accessed: 32767
134: stonewalling pairs accessed: 40556
76: stonewalling pairs accessed: 39805
763: stonewalling pairs accessed: 39735
609: stonewalling pairs accessed: 32068
577: stonewalling pairs accessed: 33864
334: stonewalling pairs accessed: 30245
12: stonewalling pairs accessed: 33762
224: stonewalling pairs accessed: 33906
91: stonewalling pairs accessed: 39872
431: stonewalling pairs accessed: 36083
380: stonewalling pairs accessed: 31456
646: stonewalling pairs accessed: 39445
633: stonewalling pairs accessed: 40302
723: stonewalling pairs accessed: 40878
253: stonewalling pairs accessed: 36123
211: stonewalling pairs accessed: 36576
473: stonewalling pairs accessed: 30826
620: stonewalling pairs accessed: 39854
68: stonewalling pairs accessed: 30761
744: stonewalling pairs accessed: 39263
49: stonewalling pairs accessed: 41172
326: stonewalling pairs accessed: 29993
1: stonewalling pairs accessed: 34763
775: stonewalling pairs accessed: 32653
89: stonewalling pairs accessed: 40348
546: stonewalling pairs accessed: 41517
506: stonewalling pairs accessed: 32517
365: stonewalling pairs accessed: 32800
373: stonewalling pairs accessed: 37311
555: stonewalling pairs accessed: 32422
614: stonewalling pairs accessed: 32025
131: stonewalling pairs accessed: 38890
73: stonewalling pairs accessed: 31962
480: stonewalling pairs accessed: 31996
477: stonewalling pairs accessed: 32324
416: stonewalling pairs accessed: 36502
417: stonewalling pairs accessed: 30787
374: stonewalling pairs accessed: 32046
791: stonewalling pairs accessed: 38672
251: stonewalling pairs accessed: 36373
47: stonewalling pairs accessed: 40772
322: stonewalling pairs accessed: 30792
713: stonewalling pairs accessed: 37922
469: stonewalling pairs accessed: 37319
397: stonewalling pairs accessed: 35057
768: stonewalling pairs accessed: 33845
357: stonewalling pairs accessed: 33488
610: stonewalling pairs accessed: 34849
135: stonewalling pairs accessed: 40358
52: stonewalling pairs accessed: 40424
240: stonewalling pairs accessed: 36468
361: stonewalling pairs accessed: 29974
625: stonewalling pairs accessed: 40329
335: stonewalling pairs accessed: 36457
430: stonewalling pairs accessed: 29921
10: stonewalling pairs accessed: 39346
379: stonewalling pairs accessed: 31906
115: stonewalling pairs accessed: 40117
536: stonewalling pairs accessed: 39440
545: stonewalling pairs accessed: 34104
561: stonewalling pairs accessed: 32965
363: stonewalling pairs accessed: 30612
782: stonewalling pairs accessed: 33943
228: stonewalling pairs accessed: 37655
92: stonewalling pairs accessed: 41000
369: stonewalling pairs accessed: 38343
544: stonewalling pairs accessed: 33262
419: stonewalling pairs accessed: 31438
617: stonewalling pairs accessed: 41429
74: stonewalling pairs accessed: 42198
297: stonewalling pairs accessed: 36122
455: stonewalling pairs accessed: 33074
329: stonewalling pairs accessed: 37125
15: stonewalling pairs accessed: 39883
488: stonewalling pairs accessed: 32631
632: stonewalling pairs accessed: 40435
527: stonewalling pairs accessed: 31362
471: stonewalling pairs accessed: 37227
770: stonewalling pairs accessed: 36373
719: stonewalling pairs accessed: 37954
381: stonewalling pairs accessed: 37991
32: stonewalling pairs accessed: 41267
668: stonewalling pairs accessed: 40141
93: stonewalling pairs accessed: 40719
66: stonewalling pairs accessed: 43255
323: stonewalling pairs accessed: 36964
359: stonewalling pairs accessed: 33697
483: stonewalling pairs accessed: 33307
436: stonewalling pairs accessed: 35955
137: stonewalling pairs accessed: 40414
565: stonewalling pairs accessed: 31862
717: stonewalling pairs accessed: 37902
424: stonewalling pairs accessed: 31416
378: stonewalling pairs accessed: 38214
580: stonewalling pairs accessed: 41940
390: stonewalling pairs accessed: 34457
519: stonewalling pairs accessed: 32387
29: stonewalling pairs accessed: 41592
542: stonewalling pairs accessed: 39802
616: stonewalling pairs accessed: 41567
767: stonewalling pairs accessed: 40480
353: stonewalling pairs accessed: 31166
494: stonewalling pairs accessed: 33454
448: stonewalling pairs accessed: 38448
574: stonewalling pairs accessed: 31804
468: stonewalling pairs accessed: 37222
530: stonewalling pairs accessed: 33815
332: stonewalling pairs accessed: 37915
779: stonewalling pairs accessed: 33834
232: stonewalling pairs accessed: 30953
677: stonewalling pairs accessed: 39101
442: stonewalling pairs accessed: 35604
571: stonewalling pairs accessed: 32428
449: stonewalling pairs accessed: 31226
540: stonewalling pairs accessed: 33228
474: stonewalling pairs accessed: 37007
624: stonewalling pairs accessed: 40725
429: stonewalling pairs accessed: 32161
250: stonewalling pairs accessed: 36433
551: stonewalling pairs accessed: 41856
17: stonewalling pairs accessed: 41339
492: stonewalling pairs accessed: 35265
756: stonewalling pairs accessed: 39834
227: stonewalling pairs accessed: 31889
658: stonewalling pairs accessed: 38624
40: stonewalling pairs accessed: 40257
568: stonewalling pairs accessed: 33381
693: stonewalling pairs accessed: 41108
549: stonewalling pairs accessed: 30287
539: stonewalling pairs accessed: 31163
427: stonewalling pairs accessed: 33238
484: stonewalling pairs accessed: 34652
709: stonewalling pairs accessed: 38354
82: stonewalling pairs accessed: 41114
526: stonewalling pairs accessed: 33948
320: stonewalling pairs accessed: 38379
362: stonewalling pairs accessed: 37027
239: stonewalling pairs accessed: 31125
456: stonewalling pairs accessed: 31829
425: stonewalling pairs accessed: 32645
541: stonewalling pairs accessed: 34059
566: stonewalling pairs accessed: 39193
490: stonewalling pairs accessed: 34865
355: stonewalling pairs accessed: 37365
226: stonewalling pairs accessed: 30777
785: stonewalling pairs accessed: 38877
451: stonewalling pairs accessed: 31223
774: stonewalling pairs accessed: 31418
553: stonewalling pairs accessed: 41030
479: stonewalling pairs accessed: 37232
513: stonewalling pairs accessed: 31335
486: stonewalling pairs accessed: 30778
537: stonewalling pairs accessed: 33996
570: stonewalling pairs accessed: 32992
707: stonewalling pairs accessed: 39002
333: stonewalling pairs accessed: 38322
354: stonewalling pairs accessed: 37085
437: stonewalling pairs accessed: 36215
516: stonewalling pairs accessed: 33854
426: stonewalling pairs accessed: 32491
548: stonewalling pairs accessed: 41660
457: stonewalling pairs accessed: 30889
660: stonewalling pairs accessed: 38634
485: stonewalling pairs accessed: 34244
584: stonewalling pairs accessed: 43360
34: stonewalling pairs accessed: 41794
534: stonewalling pairs accessed: 35144
714: stonewalling pairs accessed: 38986
700: stonewalling pairs accessed: 41265
776: stonewalling pairs accessed: 39528
605: stonewalling pairs accessed: 40033
432: stonewalling pairs accessed: 36672
236: stonewalling pairs accessed: 31391
452: stonewalling pairs accessed: 31690
554: stonewalling pairs accessed: 32616
764: stonewalling pairs accessed: 41450
567: stonewalling pairs accessed: 32661
81: stonewalling pairs accessed: 41045
771: stonewalling pairs accessed: 30074
493: stonewalling pairs accessed: 38543
525: stonewalling pairs accessed: 31371
238: stonewalling pairs accessed: 31463
366: stonewalling pairs accessed: 36957
454: stonewalling pairs accessed: 39217
144: stonewalling pairs accessed: 37239
145: stonewalling pairs accessed: 41707
146: stonewalling pairs accessed: 32444
148: stonewalling pairs accessed: 39834
149: stonewalling pairs accessed: 32470
150: stonewalling pairs accessed: 38967
151: stonewalling pairs accessed: 35606
152: stonewalling pairs accessed: 33185
153: stonewalling pairs accessed: 41367
154: stonewalling pairs accessed: 35660
155: stonewalling pairs accessed: 41470
156: stonewalling pairs accessed: 34293
157: stonewalling pairs accessed: 32256
158: stonewalling pairs accessed: 32933
159: stonewalling pairs accessed: 30483
130: stonewalling pairs accessed: 41129
762: stonewalling pairs accessed: 41478
489: stonewalling pairs accessed: 32545
547: stonewalling pairs accessed: 33961
729: stonewalling pairs accessed: 37120
518: stonewalling pairs accessed: 31695
535: stonewalling pairs accessed: 41460
450: stonewalling pairs accessed: 38316
132: stonewalling pairs accessed: 35106
559: stonewalling pairs accessed: 33750
487: stonewalling pairs accessed: 36568
726: stonewalling pairs accessed: 42253
360: stonewalling pairs accessed: 37242
233: stonewalling pairs accessed: 31780
524: stonewalling pairs accessed: 33668
783: stonewalling pairs accessed: 38855
453: stonewalling pairs accessed: 39262
13: stonewalling pairs accessed: 39807
572: stonewalling pairs accessed: 41576
230: stonewalling pairs accessed: 30539
439: stonewalling pairs accessed: 36860
772: stonewalling pairs accessed: 32078
358: stonewalling pairs accessed: 37274
520: stonewalling pairs accessed: 38454
421: stonewalling pairs accessed: 36925
481: stonewalling pairs accessed: 31033
786: stonewalling pairs accessed: 40163
460: stonewalling pairs accessed: 38985
558: stonewalling pairs accessed: 30883
563: stonewalling pairs accessed: 42059
773: stonewalling pairs accessed: 40018
515: stonewalling pairs accessed: 38672
533: stonewalling pairs accessed: 41314
462: stonewalling pairs accessed: 39047
428: stonewalling pairs accessed: 36439
237: stonewalling pairs accessed: 30705
445: stonewalling pairs accessed: 36796
552: stonewalling pairs accessed: 41659
522: stonewalling pairs accessed: 36401
458: stonewalling pairs accessed: 32319
5: stonewalling pairs accessed: 40077
443: stonewalling pairs accessed: 36498
234: stonewalling pairs accessed: 38097
556: stonewalling pairs accessed: 42075
523: stonewalling pairs accessed: 39887
420: stonewalling pairs accessed: 36268
532: stonewalling pairs accessed: 41829
459: stonewalling pairs accessed: 34689
557: stonewalling pairs accessed: 41511
573: stonewalling pairs accessed: 41514
678: stonewalling pairs accessed: 39340
777: stonewalling pairs accessed: 40501
235: stonewalling pairs accessed: 31091
423: stonewalling pairs accessed: 36615
352: stonewalling pairs accessed: 36676
507: stonewalling pairs accessed: 39106
543: stonewalling pairs accessed: 42100
780: stonewalling pairs accessed: 39648
336: stonewalling pairs accessed: 37155
338: stonewalling pairs accessed: 33479
339: stonewalling pairs accessed: 37428
340: stonewalling pairs accessed: 36969
341: stonewalling pairs accessed: 37228
343: stonewalling pairs accessed: 37366
344: stonewalling pairs accessed: 32043
345: stonewalling pairs accessed: 32899
348: stonewalling pairs accessed: 29647
349: stonewalling pairs accessed: 37113
350: stonewalling pairs accessed: 31387
351: stonewalling pairs accessed: 30123
9: stonewalling pairs accessed: 40599
418: stonewalling pairs accessed: 36484
229: stonewalling pairs accessed: 37177
466: stonewalling pairs accessed: 37773
531: stonewalling pairs accessed: 41898
769: stonewalling pairs accessed: 40273
246: stonewalling pairs accessed: 33265
645: stonewalling pairs accessed: 41711
491: stonewalling pairs accessed: 40082
778: stonewalling pairs accessed: 39944
231: stonewalling pairs accessed: 35877
33: stonewalling pairs accessed: 36307
368: stonewalling pairs accessed: 34915
8: stonewalling pairs accessed: 40450
587: stonewalling pairs accessed: 42392
79: stonewalling pairs accessed: 38484
193: stonewalling pairs accessed: 38099
618: stonewalling pairs accessed: 42788
521: stonewalling pairs accessed: 40135
65: stonewalling pairs accessed: 38207
550: stonewalling pairs accessed: 41880
512: stonewalling pairs accessed: 39977
495: stonewalling pairs accessed: 40228
325: stonewalling pairs accessed: 35422
225: stonewalling pairs accessed: 38457
511: stonewalling pairs accessed: 38683
482: stonewalling pairs accessed: 39722
463: stonewalling pairs accessed: 35814
562: stonewalling pairs accessed: 43116
stonewalling pairs accessed min: 29647 max: 44799 -- min data: 57.9 GiB mean data: 68.3 GiB time: 300.5s
WARNING: Expected aggregate file size = 8321499136000000.
WARNING: Stat() of aggregate file size = 75160249958400.
WARNING: Using actual aggregate bytes moved = 75160249958400.
WARNING: maybe caused by deadlineForStonewalling
write 160215 80230 0.000079 10158080000 2048.00 0.521854 446.71 0.427210 447.39 0
Max Write: 160214.58 MiB/sec (167997.16 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 160214.58 160214.58 160214.58 0.00 80107.29 80107.29 80107.29 0.00 447.39000 300.47 186126.88 0 800 16 1 1 1 1 0 0 1 10401873920000 2097152 71678400.0 POSIX 0
Finished : Mon Jul 6 20:51:02 2020
- ior_hard_read
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Mon Jul 6 21:43:56 2020
Command line : /opt/software/rsc/io500-2020-spring/io500-app/bin/ior -r -R -s 110000 -a MPIIO --posix.odirect -v -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard/IOR_file -O stoneWallingStatusFile=/lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard/stonewall
Machine : Linux n02p001.gvr.local
Start time skew across all tasks: 132605.10 sec
TestID : 0
StartTime : Mon Jul 6 21:43:56 2020
Path : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard
FS : 259.3 TiB Used FS: 28.0% Inodes: 1204.8 Mi Used Inodes: 16.0%
Participating tasks: 800
Using reorderTasks '-C' (useful to avoid read cache in client)
Options:
api : MPIIO
apiVersion : (3.1)
test filename : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard/IOR_file
access : single-shared-file
type : independent
segments : 110000
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 50
tasks : 800
clients per node : 16
repetitions : 1
xfersize : 47008 bytes
blocksize : 47008 bytes
aggregate filesize : 3.76 TiB
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
Commencing read performance test: Mon Jul 6 21:44:56 2020
read 11451 308850 284.23 45.91 45.91 59.06 284.93 0.836845 344.51 0
Max Read: 11451.14 MiB/sec (12007.39 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
read 11451.14 11451.14 11451.14 0.00 255432.95 255432.95 255432.95 0.00 344.51311 NA NA 0 800 16 1 0 1 1 0 0 110000 47008 47008 3945068.2 MPIIO 0
Finished : Mon Jul 6 21:49:50 2020
- ior_hard_write
-
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began : Mon Jul 6 20:57:34 2020
Command line : /opt/software/rsc/io500-2020-spring/io500-app/bin/ior -w -s 110000 -a MPIIO --posix.odirect -v -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard/IOR_file -O stoneWallingStatusFile=/lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300
Machine : Linux n02p001.gvr.local
Start time skew across all tasks: 132605.65 sec
TestID : 0
StartTime : Mon Jul 6 20:57:34 2020
Path : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard
FS : 259.3 TiB Used FS: 26.5% Inodes: 1204.8 Mi Used Inodes: 13.0%
Participating tasks: 800
Using reorderTasks '-C' (useful to avoid read cache in client)
Options:
api : MPIIO
apiVersion : (3.1)
test filename : /lustre/stor1/io500/2020.07.06-20.43.13-scr/ior_hard/IOR_file
access : single-shared-file
type : independent
segments : 110000
ordering in a file : sequential
ordering inter file : constant task offset
task offset : 1
nodes : 50
tasks : 800
clients per node : 16
repetitions : 1
xfersize : 47008 bytes
blocksize : 47008 bytes
aggregate filesize : 3.76 TiB
stonewallingTime : 300
stoneWallingWearOut : 1
Results:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---- ---------- ---------- --------- -------- -------- -------- -------- ----
Commencing write performance test: Mon Jul 6 20:58:36 2020
655: stonewalling pairs accessed: 110000
334: stonewalling pairs accessed: 110000
332: stonewalling pairs accessed: 110000
330: stonewalling pairs accessed: 110000
328: stonewalling pairs accessed: 110000
324: stonewalling pairs accessed: 110000
322: stonewalling pairs accessed: 110000
320: stonewalling pairs accessed: 110000
652: stonewalling pairs accessed: 110000
654: stonewalling pairs accessed: 110000
31: stonewalling pairs accessed: 110000
27: stonewalling pairs accessed: 110000
24: stonewalling pairs accessed: 110000
239: stonewalling pairs accessed: 110000
237: stonewalling pairs accessed: 110000
235: stonewalling pairs accessed: 110000
233: stonewalling pairs accessed: 110000
230: stonewalling pairs accessed: 110000
649: stonewalling pairs accessed: 110000
647: stonewalling pairs accessed: 110000
30: stonewalling pairs accessed: 110000
20: stonewalling pairs accessed: 110000
26: stonewalling pairs accessed: 110000
23: stonewalling pairs accessed: 110000
333: stonewalling pairs accessed: 110000
331: stonewalling pairs accessed: 110000
644: stonewalling pairs accessed: 110000
641: stonewalling pairs accessed: 110000
329: stonewalling pairs accessed: 110000
327: stonewalling pairs accessed: 110000
648: stonewalling pairs accessed: 110000
640: stonewalling pairs accessed: 110000
651: stonewalling pairs accessed: 110000
653: stonewalling pairs accessed: 110000
646: stonewalling pairs accessed: 110000
643: stonewalling pairs accessed: 110000
228: stonewalling pairs accessed: 110000
234: stonewalling pairs accessed: 110000
18: stonewalling pairs accessed: 110000
29: stonewalling pairs accessed: 110000
16: stonewalling pairs accessed: 110000
25: stonewalling pairs accessed: 110000
76: stonewalling pairs accessed: 110000
74: stonewalling pairs accessed: 110000
232: stonewalling pairs accessed: 110000
28: stonewalling pairs accessed: 110000
225: stonewalling pairs accessed: 110000
69: stonewalling pairs accessed: 110000
67: stonewalling pairs accessed: 110000
22: stonewalling pairs accessed: 110000
128: stonewalling pairs accessed: 110000
142: stonewalling pairs accessed: 110000
140: stonewalling pairs accessed: 110000
650: stonewalling pairs accessed: 110000
136: stonewalling pairs accessed: 110000
134: stonewalling pairs accessed: 110000
132: stonewalling pairs accessed: 110000
645: stonewalling pairs accessed: 110000
236: stonewalling pairs accessed: 110000
229: stonewalling pairs accessed: 110000
751: stonewalling pairs accessed: 110000
745: stonewalling pairs accessed: 110000
747: stonewalling pairs accessed: 110000
738: stonewalling pairs accessed: 110000
742: stonewalling pairs accessed: 110000
736: stonewalling pairs accessed: 110000
21: stonewalling pairs accessed: 110000
238: stonewalling pairs accessed: 110000
323: stonewalling pairs accessed: 110000
19: stonewalling pairs accessed: 110000
227: stonewalling pairs accessed: 110000
762: stonewalling pairs accessed: 110000
765: stonewalling pairs accessed: 110000
326: stonewalling pairs accessed: 110000
780: stonewalling pairs accessed: 110000
783: stonewalling pairs accessed: 110000
777: stonewalling pairs accessed: 110000
231: stonewalling pairs accessed: 110000
775: stonewalling pairs accessed: 110000
543: stonewalling pairs accessed: 110000
159: stonewalling pairs accessed: 110000
772: stonewalling pairs accessed: 110000
768: stonewalling pairs accessed: 110000
460: stonewalling pairs accessed: 110000
456: stonewalling pairs accessed: 110000
451: stonewalling pairs accessed: 110000
448: stonewalling pairs accessed: 110000
671: stonewalling pairs accessed: 110000
552: stonewalling pairs accessed: 110000
557: stonewalling pairs accessed: 110000
554: stonewalling pairs accessed: 110000
550: stonewalling pairs accessed: 110000
546: stonewalling pairs accessed: 110000
548: stonewalling pairs accessed: 110000
544: stonewalling pairs accessed: 110000
764: stonewalling pairs accessed: 110000
761: stonewalling pairs accessed: 110000
75: stonewalling pairs accessed: 110000
66: stonewalling pairs accessed: 110000
17: stonewalling pairs accessed: 110000
799: stonewalling pairs accessed: 110000
325: stonewalling pairs accessed: 110000
623: stonewalling pairs accessed: 110000
619: stonewalling pairs accessed: 110000
616: stonewalling pairs accessed: 110000
613: stonewalling pairs accessed: 110000
73: stonewalling pairs accessed: 110000
610: stonewalling pairs accessed: 110000
608: stonewalling pairs accessed: 110000
759: stonewalling pairs accessed: 110000
767: stonewalling pairs accessed: 110000
68: stonewalling pairs accessed: 110000
46: stonewalling pairs accessed: 110000
44: stonewalling pairs accessed: 110000
287: stonewalling pairs accessed: 110000
542: stonewalling pairs accessed: 110000
540: stonewalling pairs accessed: 110000
538: stonewalling pairs accessed: 110000
536: stonewalling pairs accessed: 110000
669: stonewalling pairs accessed: 110000
666: stonewalling pairs accessed: 110000
686: stonewalling pairs accessed: 110000
679: stonewalling pairs accessed: 110000
683: stonewalling pairs accessed: 110000
676: stonewalling pairs accessed: 110000
674: stonewalling pairs accessed: 110000
672: stonewalling pairs accessed: 110000
763: stonewalling pairs accessed: 110000
758: stonewalling pairs accessed: 110000
760: stonewalling pairs accessed: 110000
756: stonewalling pairs accessed: 110000
154: stonewalling pairs accessed: 110000
156: stonewalling pairs accessed: 110000
639: stonewalling pairs accessed: 110000
72: stonewalling pairs accessed: 110000
63: stonewalling pairs accessed: 110000
351: stonewalling pairs accessed: 110000
479: stonewalling pairs accessed: 110000
321: stonewalling pairs accessed: 110000
152: stonewalling pairs accessed: 110000
150: stonewalling pairs accessed: 110000
766: stonewalling pairs accessed: 110000
42: stonewalling pairs accessed: 110000
663: stonewalling pairs accessed: 110000
658: stonewalling pairs accessed: 110000
635: stonewalling pairs accessed: 110000
632: stonewalling pairs accessed: 110000
71: stonewalling pairs accessed: 110000
38: stonewalling pairs accessed: 110000
33: stonewalling pairs accessed: 110000
319: stonewalling pairs accessed: 110000
642: stonewalling pairs accessed: 110000
45: stonewalling pairs accessed: 110000
755: stonewalling pairs accessed: 110000
702: stonewalling pairs accessed: 110000
753: stonewalling pairs accessed: 110000
757: stonewalling pairs accessed: 110000
70: stonewalling pairs accessed: 110000
476: stonewalling pairs accessed: 110000
147: stonewalling pairs accessed: 110000
144: stonewalling pairs accessed: 110000
696: stonewalling pairs accessed: 110000
43: stonewalling pairs accessed: 110000
95: stonewalling pairs accessed: 110000
694: stonewalling pairs accessed: 110000
691: stonewalling pairs accessed: 110000
255: stonewalling pairs accessed: 110000
447: stonewalling pairs accessed: 110000
719: stonewalling pairs accessed: 110000
713: stonewalling pairs accessed: 110000
710: stonewalling pairs accessed: 110000
715: stonewalling pairs accessed: 110000
478: stonewalling pairs accessed: 110000
707: stonewalling pairs accessed: 110000
704: stonewalling pairs accessed: 110000
688: stonewalling pairs accessed: 110000
629: stonewalling pairs accessed: 110000
32: stonewalling pairs accessed: 110000
541: stonewalling pairs accessed: 110000
575: stonewalling pairs accessed: 110000
141: stonewalling pairs accessed: 110000
135: stonewalling pairs accessed: 110000
491: stonewalling pairs accessed: 110000
493: stonewalling pairs accessed: 110000
286: stonewalling pairs accessed: 110000
477: stonewalling pairs accessed: 110000
139: stonewalling pairs accessed: 110000
735: stonewalling pairs accessed: 110000
127: stonewalling pairs accessed: 110000
151: stonewalling pairs accessed: 110000
539: stonewalling pairs accessed: 110000
537: stonewalling pairs accessed: 110000
627: stonewalling pairs accessed: 110000
58: stonewalling pairs accessed: 110000
730: stonewalling pairs accessed: 110000
727: stonewalling pairs accessed: 110000
703: stonewalling pairs accessed: 110000
62: stonewalling pairs accessed: 110000
638: stonewalling pairs accessed: 110000
41: stonewalling pairs accessed: 110000
489: stonewalling pairs accessed: 110000
284: stonewalling pairs accessed: 110000
487: stonewalling pairs accessed: 110000
634: stonewalling pairs accessed: 110000
15: stonewalling pairs accessed: 110000
11: stonewalling pairs accessed: 110000
475: stonewalling pairs accessed: 110000
278: stonewalling pairs accessed: 110000
670: stonewalling pairs accessed: 110000
668: stonewalling pairs accessed: 110000
65: stonewalling pairs accessed: 110000
223: stonewalling pairs accessed: 110000
631: stonewalling pairs accessed: 110000
750: stonewalling pairs accessed: 110000
746: stonewalling pairs accessed: 110000
149: stonewalling pairs accessed: 110000
146: stonewalling pairs accessed: 110000
794: stonewalling pairs accessed: 110000
474: stonewalling pairs accessed: 110000
798: stonewalling pairs accessed: 110000
790: stonewalling pairs accessed: 110000
37: stonewalling pairs accessed: 110000
61: stonewalling pairs accessed: 110000
57: stonewalling pairs accessed: 110000
469: stonewalling pairs accessed: 110000
628: stonewalling pairs accessed: 110000
626: stonewalling pairs accessed: 110000
158: stonewalling pairs accessed: 110000
155: stonewalling pairs accessed: 110000
254: stonewalling pairs accessed: 110000
251: stonewalling pairs accessed: 110000
445: stonewalling pairs accessed: 110000
443: stonewalling pairs accessed: 110000
138: stonewalling pairs accessed: 110000
285: stonewalling pairs accessed: 110000
787: stonewalling pairs accessed: 110000
473: stonewalling pairs accessed: 110000
163: stonewalling pairs accessed: 110000
172: stonewalling pairs accessed: 110000
160: stonewalling pairs accessed: 110000
165: stonewalling pairs accessed: 110000
168: stonewalling pairs accessed: 110000
174: stonewalling pairs accessed: 110000
64: stonewalling pairs accessed: 110000
224: stonewalling pairs accessed: 110000
793: stonewalling pairs accessed: 110000
637: stonewalling pairs accessed: 110000
786: stonewalling pairs accessed: 110000
55: stonewalling pairs accessed: 110000
789: stonewalling pairs accessed: 110000
350: stonewalling pairs accessed: 110000
348: stonewalling pairs accessed: 110000
226: stonewalling pairs accessed: 110000
315: stonewalling pairs accessed: 110000
556: stonewalling pairs accessed: 110000
283: stonewalling pairs accessed: 110000
553: stonewalling pairs accessed: 110000
741: stonewalling pairs accessed: 110000
726: stonewalling pairs accessed: 110000
744: stonewalling pairs accessed: 110000
133: stonewalling pairs accessed: 110000
734: stonewalling pairs accessed: 110000
8: stonewalling pairs accessed: 110000
693: stonewalling pairs accessed: 110000
143: stonewalling pairs accessed: 110000
551: stonewalling pairs accessed: 110000
40: stonewalling pairs accessed: 110000
36: stonewalling pairs accessed: 110000
701: stonewalling pairs accessed: 110000
685: stonewalling pairs accessed: 110000
153: stonewalling pairs accessed: 110000
472: stonewalling pairs accessed: 110000
682: stonewalling pairs accessed: 110000
246: stonewalling pairs accessed: 110000
248: stonewalling pairs accessed: 110000
754: stonewalling pairs accessed: 110000
797: stonewalling pairs accessed: 110000
188: stonewalling pairs accessed: 110000
186: stonewalling pairs accessed: 110000
729: stonewalling pairs accessed: 110000
511: stonewalling pairs accessed: 110000
124: stonewalling pairs accessed: 110000
122: stonewalling pairs accessed: 110000
346: stonewalling pairs accessed: 110000
194: stonewalling pairs accessed: 110000
192: stonewalling pairs accessed: 110000
318: stonewalling pairs accessed: 110000
196: stonewalling pairs accessed: 110000
202: stonewalling pairs accessed: 110000
792: stonewalling pairs accessed: 110000
488: stonewalling pairs accessed: 110000
282: stonewalling pairs accessed: 110000
607: stonewalling pairs accessed: 110000
572: stonewalling pairs accessed: 110000
570: stonewalling pairs accessed: 110000
344: stonewalling pairs accessed: 110000
695: stonewalling pairs accessed: 110000
191: stonewalling pairs accessed: 110000
549: stonewalling pairs accessed: 110000
484: stonewalling pairs accessed: 110000
486: stonewalling pairs accessed: 110000
471: stonewalling pairs accessed: 110000
796: stonewalling pairs accessed: 110000
241: stonewalling pairs accessed: 110000
94: stonewalling pairs accessed: 110000
90: stonewalling pairs accessed: 110000
311: stonewalling pairs accessed: 110000
314: stonewalling pairs accessed: 110000
281: stonewalling pairs accessed: 110000
665: stonewalling pairs accessed: 110000
39: stonewalling pairs accessed: 110000
566: stonewalling pairs accessed: 110000
563: stonewalling pairs accessed: 110000
791: stonewalling pairs accessed: 110000
788: stonewalling pairs accessed: 110000
785: stonewalling pairs accessed: 110000
530: stonewalling pairs accessed: 110000
535: stonewalling pairs accessed: 110000
662: stonewalling pairs accessed: 110000
657: stonewalling pairs accessed: 110000
733: stonewalling pairs accessed: 110000
725: stonewalling pairs accessed: 110000
728: stonewalling pairs accessed: 110000
277: stonewalling pairs accessed: 110000
60: stonewalling pairs accessed: 110000
440: stonewalling pairs accessed: 110000
157: stonewalling pairs accessed: 110000
624: stonewalling pairs accessed: 110000
468: stonewalling pairs accessed: 110000
465: stonewalling pairs accessed: 110000
317: stonewalling pairs accessed: 110000
5: stonewalling pairs accessed: 110000
313: stonewalling pairs accessed: 110000
700: stonewalling pairs accessed: 110000
1: stonewalling pairs accessed: 110000
339: stonewalling pairs accessed: 110000
253: stonewalling pairs accessed: 110000
250: stonewalling pairs accessed: 110000
470: stonewalling pairs accessed: 110000
492: stonewalling pairs accessed: 110000
490: stonewalling pairs accessed: 110000
574: stonewalling pairs accessed: 110000
247: stonewalling pairs accessed: 110000
483: stonewalling pairs accessed: 110000
52: stonewalling pairs accessed: 110000
245: stonewalling pairs accessed: 110000
88: stonewalling pairs accessed: 110000
743: stonewalling pairs accessed: 110000
749: stonewalling pairs accessed: 110000
302: stonewalling pairs accessed: 110000
190: stonewalling pairs accessed: 110000
297: stonewalling pairs accessed: 110000
434: stonewalling pairs accessed: 110000
35: stonewalling pairs accessed: 110000
50: stonewalling pairs accessed: 110000
56: stonewalling pairs accessed: 110000
782: stonewalling pairs accessed: 110000
779: stonewalling pairs accessed: 110000
442: stonewalling pairs accessed: 110000
10: stonewalling pairs accessed: 110000
547: stonewalling pairs accessed: 110000
545: stonewalling pairs accessed: 110000
732: stonewalling pairs accessed: 110000
338: stonewalling pairs accessed: 110000
137: stonewalling pairs accessed: 110000
622: stonewalling pairs accessed: 110000
618: stonewalling pairs accessed: 110000
187: stonewalling pairs accessed: 110000
349: stonewalling pairs accessed: 110000
347: stonewalling pairs accessed: 110000
185: stonewalling pairs accessed: 110000
180: stonewalling pairs accessed: 110000
506: stonewalling pairs accessed: 110000
504: stonewalling pairs accessed: 110000
502: stonewalling pairs accessed: 110000
93: stonewalling pairs accessed: 110000
724: stonewalling pairs accessed: 110000
85: stonewalling pairs accessed: 110000
721: stonewalling pairs accessed: 110000
54: stonewalling pairs accessed: 110000
776: stonewalling pairs accessed: 110000
345: stonewalling pairs accessed: 110000
441: stonewalling pairs accessed: 110000
444: stonewalling pairs accessed: 110000
774: stonewalling pairs accessed: 110000
268: stonewalling pairs accessed: 110000
591: stonewalling pairs accessed: 110000
625: stonewalling pairs accessed: 110000
588: stonewalling pairs accessed: 110000
362: stonewalling pairs accessed: 110000
383: stonewalling pairs accessed: 110000
343: stonewalling pairs accessed: 110000
365: stonewalling pairs accessed: 110000
439: stonewalling pairs accessed: 110000
699: stonewalling pairs accessed: 110000
131: stonewalling pairs accessed: 110000
467: stonewalling pairs accessed: 110000
189: stonewalling pairs accessed: 110000
265: stonewalling pairs accessed: 110000
222: stonewalling pairs accessed: 110000
219: stonewalling pairs accessed: 110000
0: stonewalling pairs accessed: 110000
571: stonewalling pairs accessed: 110000
569: stonewalling pairs accessed: 110000
633: stonewalling pairs accessed: 110000
34: stonewalling pairs accessed: 110000
481: stonewalling pairs accessed: 110000
51: stonewalling pairs accessed: 110000
7: stonewalling pairs accessed: 110000
496: stonewalling pairs accessed: 110000
49: stonewalling pairs accessed: 110000
89: stonewalling pairs accessed: 110000
529: stonewalling pairs accessed: 110000
87: stonewalling pairs accessed: 110000
84: stonewalling pairs accessed: 110000
459: stonewalling pairs accessed: 110000
621: stonewalling pairs accessed: 110000
455: stonewalling pairs accessed: 110000
450: stonewalling pairs accessed: 110000
59: stonewalling pairs accessed: 110000
291: stonewalling pairs accessed: 110000
397: stonewalling pairs accessed: 110000
114: stonewalling pairs accessed: 110000
112: stonewalling pairs accessed: 110000
415: stonewalling pairs accessed: 110000
109: stonewalling pairs accessed: 110000
752: stonewalling pairs accessed: 110000
795: stonewalling pairs accessed: 110000
104: stonewalling pairs accessed: 110000
604: stonewalling pairs accessed: 110000
598: stonewalling pairs accessed: 110000
396: stonewalling pairs accessed: 110000
580: stonewalling pairs accessed: 110000
534: stonewalling pairs accessed: 110000
576: stonewalling pairs accessed: 110000
582: stonewalling pairs accessed: 110000
586: stonewalling pairs accessed: 110000
485: stonewalling pairs accessed: 110000
261: stonewalling pairs accessed: 110000
630: stonewalling pairs accessed: 110000
594: stonewalling pairs accessed: 110000
656: stonewalling pairs accessed: 110000
217: stonewalling pairs accessed: 110000
296: stonewalling pairs accessed: 110000
364: stonewalling pairs accessed: 110000
342: stonewalling pairs accessed: 110000
429: stonewalling pairs accessed: 110000
361: stonewalling pairs accessed: 110000
184: stonewalling pairs accessed: 110000
731: stonewalling pairs accessed: 110000
252: stonewalling pairs accessed: 110000
249: stonewalling pairs accessed: 110000
723: stonewalling pairs accessed: 110000
9: stonewalling pairs accessed: 110000
664: stonewalling pairs accessed: 110000
4: stonewalling pairs accessed: 110000
771: stonewalling pairs accessed: 110000
565: stonewalling pairs accessed: 110000
573: stonewalling pairs accessed: 110000
310: stonewalling pairs accessed: 110000
480: stonewalling pairs accessed: 110000
698: stonewalling pairs accessed: 110000
781: stonewalling pairs accessed: 110000
164: stonewalling pairs accessed: 110000
510: stonewalling pairs accessed: 110000
360: stonewalling pairs accessed: 110000
264: stonewalling pairs accessed: 110000
363: stonewalling pairs accessed: 110000
53: stonewalling pairs accessed: 110000
267: stonewalling pairs accessed: 110000
386: stonewalling pairs accessed: 110000
80: stonewalling pairs accessed: 110000
636: stonewalling pairs accessed: 110000
378: stonewalling pairs accessed: 110000
373: stonewalling pairs accessed: 110000
258: stonewalling pairs accessed: 110000
301: stonewalling pairs accessed: 110000
260: stonewalling pairs accessed: 110000
426: stonewalling pairs accessed: 110000
617: stonewalling pairs accessed: 110000
615: stonewalling pairs accessed: 110000
312: stonewalling pairs accessed: 110000
678: stonewalling pairs accessed: 110000
183: stonewalling pairs accessed: 110000
454: stonewalling pairs accessed: 110000
395: stonewalling pairs accessed: 110000
505: stonewalling pairs accessed: 110000
221: stonewalling pairs accessed: 110000
280: stonewalling pairs accessed: 110000
295: stonewalling pairs accessed: 110000
290: stonewalling pairs accessed: 110000
279: stonewalling pairs accessed: 110000
126: stonewalling pairs accessed: 110000
309: stonewalling pairs accessed: 110000
305: stonewalling pairs accessed: 110000
337: stonewalling pairs accessed: 110000
79: stonewalling pairs accessed: 110000
276: stonewalling pairs accessed: 110000
714: stonewalling pairs accessed: 110000
718: stonewalling pairs accessed: 110000
661: stonewalling pairs accessed: 110000
562: stonewalling pairs accessed: 110000
568: stonewalling pairs accessed: 110000
341: stonewalling pairs accessed: 110000
684: stonewalling pairs accessed: 110000
6: stonewalling pairs accessed: 110000
446: stonewalling pairs accessed: 110000
3: stonewalling pairs accessed: 110000
712: stonewalling pairs accessed: 110000
14: stonewalling pairs accessed: 110000
709: stonewalling pairs accessed: 110000
778: stonewalling pairs accessed: 110000
466: stonewalling pairs accessed: 110000
464: stonewalling pairs accessed: 110000
533: stonewalling pairs accessed: 110000
528: stonewalling pairs accessed: 110000
748: stonewalling pairs accessed: 110000
495: stonewalling pairs accessed: 110000
394: stonewalling pairs accessed: 110000
300: stonewalling pairs accessed: 110000
503: stonewalling pairs accessed: 110000
438: stonewalling pairs accessed: 110000
697: stonewalling pairs accessed: 110000
681: stonewalling pairs accessed: 110000
294: stonewalling pairs accessed: 110000
677: stonewalling pairs accessed: 110000
675: stonewalling pairs accessed: 110000
509: stonewalling pairs accessed: 110000
316: stonewalling pairs accessed: 110000
308: stonewalling pairs accessed: 110000
687: stonewalling pairs accessed: 110000
218: stonewalling pairs accessed: 110000
216: stonewalling pairs accessed: 110000
667: stonewalling pairs accessed: 110000
359: stonewalling pairs accessed: 110000
92: stonewalling pairs accessed: 110000
123: stonewalling pairs accessed: 110000
266: stonewalling pairs accessed: 110000
433: stonewalling pairs accessed: 110000
121: stonewalling pairs accessed: 110000
78: stonewalling pairs accessed: 110000
393: stonewalling pairs accessed: 110000
458: stonewalling pairs accessed: 110000
453: stonewalling pairs accessed: 110000
449: stonewalling pairs accessed: 110000
108: stonewalling pairs accessed: 110000
293: stonewalling pairs accessed: 110000
706: stonewalling pairs accessed: 110000
299: stonewalling pairs accessed: 110000
690: stonewalling pairs accessed: 110000
692: stonewalling pairs accessed: 110000
567: stonewalling pairs accessed: 110000
358: stonewalling pairs accessed: 110000
620: stonewalling pairs accessed: 110000
501: stonewalling pairs accessed: 110000
555: stonewalling pairs accessed: 110000
740: stonewalling pairs accessed: 110000
173: stonewalling pairs accessed: 110000
171: stonewalling pairs accessed: 110000
292: stonewalling pairs accessed: 110000
263: stonewalling pairs accessed: 110000
508: stonewalling pairs accessed: 110000
275: stonewalling pairs accessed: 110000
209: stonewalling pairs accessed: 110000
257: stonewalling pairs accessed: 110000
259: stonewalling pairs accessed: 110000
13: stonewalling pairs accessed: 110000
392: stonewalling pairs accessed: 110000
2: stonewalling pairs accessed: 110000
307: stonewalling pairs accessed: 110000
148: stonewalling pairs accessed: 110000
391: stonewalling pairs accessed: 110000
660: stonewalling pairs accessed: 110000
452: stonewalling pairs accessed: 110000
717: stonewalling pairs accessed: 110000
145: stonewalling pairs accessed: 110000
289: stonewalling pairs accessed: 110000
240: stonewalling pairs accessed: 110000
357: stonewalling pairs accessed: 110000
298: stonewalling pairs accessed: 110000
385: stonewalling pairs accessed: 110000
597: stonewalling pairs accessed: 110000
711: stonewalling pairs accessed: 110000
244: stonewalling pairs accessed: 110000
482: stonewalling pairs accessed: 110000
722: stonewalling pairs accessed: 110000
77: stonewalling pairs accessed: 110000
522: stonewalling pairs accessed: 110000
524: stonewalling pairs accessed: 110000
179: stonewalling pairs accessed: 110000
48: stonewalling pairs accessed: 110000
120: stonewalling pairs accessed: 110000
220: stonewalling pairs accessed: 110000
256: stonewalling pairs accessed: 110000
274: stonewalling pairs accessed: 110000
414: stonewalling pairs accessed: 110000
680: stonewalling pairs accessed: 110000
384: stonewalling pairs accessed: 110000
390: stonewalling pairs accessed: 110000
412: stonewalling pairs accessed: 110000
304: stonewalling pairs accessed: 110000
288: stonewalling pairs accessed: 110000
673: stonewalling pairs accessed: 110000
457: stonewalling pairs accessed: 110000
428: stonewalling pairs accessed: 110000
437: stonewalling pairs accessed: 110000
494: stonewalling pairs accessed: 110000
770: stonewalling pairs accessed: 110000
125: stonewalling pairs accessed: 110000
773: stonewalling pairs accessed: 110000
564: stonewalling pairs accessed: 110000
527: stonewalling pairs accessed: 110000
425: stonewalling pairs accessed: 110000
389: stonewalling pairs accessed: 110000
91: stonewalling pairs accessed: 110000
182: stonewalling pairs accessed: 110000
603: stonewalling pairs accessed: 110000
356: stonewalling pairs accessed: 110000
86: stonewalling pairs accessed: 110000
561: stonewalling pairs accessed: 110000
262: stonewalling pairs accessed: 110000
612: stonewalling pairs accessed: 110000
336: stonewalling pairs accessed: 110000
432: stonewalling pairs accessed: 110000
97: stonewalling pairs accessed: 110000
103: stonewalling pairs accessed: 110000
689: stonewalling pairs accessed: 110000
215: stonewalling pairs accessed: 110000
12: stonewalling pairs accessed: 110000
107: stonewalling pairs accessed: 110000
178: stonewalling pairs accessed: 110000
83: stonewalling pairs accessed: 110000
273: stonewalling pairs accessed: 110000
388: stonewalling pairs accessed: 110000
355: stonewalling pairs accessed: 110000
243: stonewalling pairs accessed: 110000
382: stonewalling pairs accessed: 110000
784: stonewalling pairs accessed: 110000
377: stonewalling pairs accessed: 110000
602: stonewalling pairs accessed: 110000
606: stonewalling pairs accessed: 110000
372: stonewalling pairs accessed: 110000
130: stonewalling pairs accessed: 110000
420: stonewalling pairs accessed: 110000
102: stonewalling pairs accessed: 110000
96: stonewalling pairs accessed: 110000
532: stonewalling pairs accessed: 110000
737: stonewalling pairs accessed: 110000
306: stonewalling pairs accessed: 110000
708: stonewalling pairs accessed: 110000
614: stonewalling pairs accessed: 110000
201: stonewalling pairs accessed: 110000
193: stonewalling pairs accessed: 110000
354: stonewalling pairs accessed: 110000
659: stonewalling pairs accessed: 110000
500: stonewalling pairs accessed: 110000
47: stonewalling pairs accessed: 110000
195: stonewalling pairs accessed: 110000
407: stonewalling pairs accessed: 110000
387: stonewalling pairs accessed: 110000
716: stonewalling pairs accessed: 110000
101: stonewalling pairs accessed: 110000
106: stonewalling pairs accessed: 110000
611: stonewalling pairs accessed: 110000
705: stonewalling pairs accessed: 110000
609: stonewalling pairs accessed: 110000
507: stonewalling pairs accessed: 110000
526: stonewalling pairs accessed: 110000
560: stonewalling pairs accessed: 110000
427: stonewalling pairs accessed: 110000
272: stonewalling pairs accessed: 110000
111: stonewalling pairs accessed: 110000
303: stonewalling pairs accessed: 110000
381: stonewalling pairs accessed: 110000
376: stonewalling pairs accessed: 110000
105: stonewalling pairs accessed: 110000
413: stonewalling pairs accessed: 110000
424: stonewalling pairs accessed: 110000
371: stonewalling pairs accessed: 110000
335: stonewalling pairs accessed: 110000
170: stonewalling pairs accessed: 110000
242: stonewalling pairs accessed: 110000
499: stonewalling pairs accessed: 110000
353: stonewalling pairs accessed: 110000
400: stonewalling pairs accessed: 110000
100: stonewalling pairs accessed: 110000
739: stonewalling pairs accessed: 110000
352: stonewalling pairs accessed: 110000
463: stonewalling pairs accessed: 110000
419: stonewalling pairs accessed: 110000
82: stonewalling pairs accessed: 110000
110: stonewalling pairs accessed: 110000
113: stonewalling pairs accessed: 110000
200: stonewalling pairs accessed: 110000
590: stonewalling pairs accessed: 110000
181: stonewalling pairs accessed: 110000
720: stonewalling pairs accessed: 110000
177: stonewalling pairs accessed: 110000
167: stonewalling pairs accessed: 110000
208: stonewalling pairs accessed: 110000
380: stonewalling pairs accessed: 110000
375: stonewalling pairs accessed: 110000
531: stonewalling pairs accessed: 110000
271: stonewalling pairs accessed: 110000
119: stonewalling pairs accessed: 110000
340: stonewalling pairs accessed: 110000
525: stonewalling pairs accessed: 110000
559: stonewalling pairs accessed: 110000
411: stonewalling pairs accessed: 110000
369: stonewalling pairs accessed: 110000
605: stonewalling pairs accessed: 110000
169: stonewalling pairs accessed: 110000
162: stonewalling pairs accessed: 110000
587: stonewalling pairs accessed: 110000
431: stonewalling pairs accessed: 110000
462: stonewalling pairs accessed: 110000
423: stonewalling pairs accessed: 110000
270: stonewalling pairs accessed: 110000
399: stonewalling pairs accessed: 110000
379: stonewalling pairs accessed: 110000
207: stonewalling pairs accessed: 110000
118: stonewalling pairs accessed: 110000
498: stonewalling pairs accessed: 110000
406: stonewalling pairs accessed: 110000
601: stonewalling pairs accessed: 110000
374: stonewalling pairs accessed: 110000
418: stonewalling pairs accessed: 110000
81: stonewalling pairs accessed: 110000
368: stonewalling pairs accessed: 110000
585: stonewalling pairs accessed: 110000
518: stonewalling pairs accessed: 110000
166: stonewalling pairs accessed: 110000
199: stonewalling pairs accessed: 110000
558: stonewalling pairs accessed: 110000
214: stonewalling pairs accessed: 110000
206: stonewalling pairs accessed: 110000
176: stonewalling pairs accessed: 110000
769: stonewalling pairs accessed: 110000
205: stonewalling pairs accessed: 110000
398: stonewalling pairs accessed: 110000
99: stonewalling pairs accessed: 110000
581: stonewalling pairs accessed: 110000
579: stonewalling pairs accessed: 110000
410: stonewalling pairs accessed: 110000
430: stonewalling pairs accessed: 110000
422: stonewalling pairs accessed: 110000
516: stonewalling pairs accessed: 110000
117: stonewalling pairs accessed: 110000
461: stonewalling pairs accessed: 110000
436: stonewalling pairs accessed: 110000
198: stonewalling pairs accessed: 110000
405: stonewalling pairs accessed: 110000
213: stonewalling pairs accessed: 110000
204: stonewalling pairs accessed: 110000
589: stonewalling pairs accessed: 110000
98: stonewalling pairs accessed: 110000
523: stonewalling pairs accessed: 110000
161: stonewalling pairs accessed: 110000
175: stonewalling pairs accessed: 110000
404: stonewalling pairs accessed: 110000
269: stonewalling pairs accessed: 110000
116: stonewalling pairs accessed: 110000
596: stonewalling pairs accessed: 110000
593: stonewalling pairs accessed: 110000
497: stonewalling pairs accessed: 110000
367: stonewalling pairs accessed: 110000
409: stonewalling pairs accessed: 110000
513: stonewalling pairs accessed: 110000
584: stonewalling pairs accessed: 110000
600: stonewalling pairs accessed: 110000
595: stonewalling pairs accessed: 110000
417: stonewalling pairs accessed: 110000
521: stonewalling pairs accessed: 110000
212: stonewalling pairs accessed: 110000
408: stonewalling pairs accessed: 110000
197: stonewalling pairs accessed: 110000
203: stonewalling pairs accessed: 110000
421: stonewalling pairs accessed: 110000
403: stonewalling pairs accessed: 110000
115: stonewalling pairs accessed: 110000
416: stonewalling pairs accessed: 110000
592: stonewalling pairs accessed: 110000
578: stonewalling pairs accessed: 110000
583: stonewalling pairs accessed: 110000
402: stonewalling pairs accessed: 110000
366: stonewalling pairs accessed: 110000
129: stonewalling pairs accessed: 110000
435: stonewalling pairs accessed: 110000
512: stonewalling pairs accessed: 110000
370: stonewalling pairs accessed: 110000
599: stonewalling pairs accessed: 110000
211: stonewalling pairs accessed: 110000
517: stonewalling pairs accessed: 110000
210: stonewalling pairs accessed: 110000
577: stonewalling pairs accessed: 110000
515: stonewalling pairs accessed: 110000
520: stonewalling pairs accessed: 110000
401: stonewalling pairs accessed: 110000
514: stonewalling pairs accessed: 110000
519: stonewalling pairs accessed: 110000
stonewalling pairs accessed min: 110000 max: 110000 -- min data: 4.8 GiB mean data: 4.8 GiB time: 268.8s
write 11946 326401 269.26 45.91 45.91 59.97 269.61 0.998167 330.23 0
Max Write: 11946.37 MiB/sec (12526.68 MB/sec)
Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Max(OPs) Min(OPs) Mean(OPs) StdDev Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggs(MiB) API RefNum
write 11946.37 11946.37 11946.37 0.00 266479.71 266479.71 266479.71 0.00 330.23152 268.76 14678.63 0 800 16 1 0 1 1 0 0 110000 47008 47008 3945068.2 MPIIO 0
Finished : Mon Jul 6 21:03:14 2020
- mdtest_easy_delete
-
-- started at 07/06/2020 21:53:39 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-r' '-F' '-P' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_easy' '-n' '1000000' '-u' '-L' '-a' 'POSIX' '--posix.odirect' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_easy-stonewall' '-N' '1'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 28.0% Inodes: 1204.8 Mi Used Inodes: 16.0%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 348722.235 348469.747 348603.307 51.270
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.739 0.739 0.739 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 472.307 471.965 472.126 0.069
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 1.354 1.354 1.354 0.000
-- finished at 07/06/2020 22:02:34 --
- mdtest_easy_stat
-
-- started at 07/06/2020 21:40:22 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-T' '-F' '-P' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_easy' '-n' '1000000' '-u' '-L' '-a' 'POSIX' '--posix.odirect' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_easy-stonewall' '-N' '1'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 28.0% Inodes: 1204.8 Mi Used Inodes: 16.0%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 858774.717 856814.417 857693.766 403.759
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 192.089 191.651 191.892 0.090
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/06/2020 21:43:39 --
- mdtest_easy_write
-
-- started at 07/06/2020 20:51:19 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-Y' '-C' '-F' '-P' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_easy' '-n' '1000000' '-u' '-L' '-a' 'POSIX' '--posix.odirect' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_easy-stonewall' '-N' '1' '-W' '300'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 26.5% Inodes: 1047.8 Mi Used Inodes: 0.0%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
Continue stonewall hit min: 189301 max: 205731 avg: 195334.1
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 492381.913 489886.502 492079.820 203.836
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 523695.280 NA
Tree creation : 1.259 1.259 1.259 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 335.965 334.262 334.468 0.139
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 298.394 NA
Tree creation : 0.794 0.794 0.794 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/06/2020 20:57:18 --
- mdtest_hard_delete
-
-- started at 07/06/2020 22:06:58 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-r' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 28.0% Inodes: 1047.8 Mi Used Inodes: 3.4%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 67448.844 67408.665 67434.232 7.666
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.094 0.094 0.094 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 554.991 554.660 554.781 0.063
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 10.640 10.640 10.640 0.000
-- finished at 07/06/2020 22:16:29 --
- mdtest_hard_read
-
-- started at 07/06/2020 22:02:50 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-X' '-E' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 28.0% Inodes: 1047.8 Mi Used Inodes: 3.4%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 164740.751 164371.959 164591.125 58.189
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 0.000 0.000 0.000 0.000
File read : 227.601 227.091 227.298 0.080
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/06/2020 22:06:43 --
- mdtest_hard_stat
-
-- started at 07/06/2020 21:50:11 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-T' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 28.0% Inodes: 1204.8 Mi Used Inodes: 16.0%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 201261.283 200921.820 201094.047 65.050
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 0.000 0.000 0.000 0.000
File stat : 186.198 185.884 186.038 0.060
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
Tree creation : 0.000 0.000 0.000 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/06/2020 21:53:23 --
- mdtest_hard_write
-
-- started at 07/06/2020 21:03:30 --
mdtest-3.3.0+dev was launched with 800 total task(s) on 50 node(s)
Command line used: /opt/software/rsc/io500-2020-spring/io500-app/bin/mdtest '-Y' '-C' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/stor1/io500/2020.07.06-20.43.13-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1' '-W' '300'
Path: /lustre/stor1/io500/2020.07.06-20.43.13-scr
FS: 259.3 TiB Used FS: 27.9% Inodes: 1204.8 Mi Used Inodes: 13.0%
Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank 0 Line 2166 Shifting ranks by 16 for each phase.
800 tasks, 800000000 files
Continue stonewall hit min: 44107 max: 46764 avg: 45235.4
SUMMARY rate: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 113767.137 113654.040 113727.375 23.756
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 121277.964 NA
Tree creation : 1.091 1.091 1.091 0.000
Tree removal : 0.000 0.000 0.000 0.000
SUMMARY time: (of 1 iterations)
Operation Max Min Mean Std Dev
--------- --- --- ---- -------
File creation : 329.167 328.840 328.955 0.069
File stat : 0.000 0.000 0.000 0.000
File read : 0.000 0.000 0.000 0.000
File removal : 0.000 0.000 0.000 0.000
File create (stonewall) : NA NA 298.391 NA
Tree creation : 0.917 0.917 0.917 0.000
Tree removal : 0.000 0.000 0.000 0.000
-- finished at 07/06/2020 21:09:48 --
- result_summary
-
[RESULT] BW phase 1 ior_easy_write 156.460 GiB/s : time 446.71 seconds
[RESULT] IOPS phase 1 mdtest_easy_write 492.382 kiops : time 335.96 seconds
[RESULT] BW phase 2 ior_hard_write 11.666 GiB/s : time 269.61 seconds
[RESULT] IOPS phase 2 mdtest_hard_write 113.767 kiops : time 329.17 seconds
[RESULT] IOPS phase 3 find 222.760 kiops : time 906.78 seconds
[RESULT] BW phase 3 ior_easy_read 78.783 GiB/s : time 887.59 seconds
[RESULT] IOPS phase 4 mdtest_easy_stat 858.775 kiops : time 192.09 seconds
[RESULT] BW phase 4 ior_hard_read 11.183 GiB/s : time 284.93 seconds
[RESULT] IOPS phase 5 mdtest_hard_stat 201.261 kiops : time 186.20 seconds
[RESULT] IOPS phase 6 mdtest_easy_delete 348.722 kiops : time 472.31 seconds
[RESULT] IOPS phase 7 mdtest_hard_read 164.741 kiops : time 227.60 seconds
[RESULT] IOPS phase 8 mdtest_hard_delete 67.449 kiops : time 586.53 seconds
[SCORE] Bandwidth 35.6103 GiB/s : IOPS 231.876 kiops : TOTAL 90.869