Polytechnic RSC Tornado

Institution SPbPU
Client Procs Per Node
Client Operating System CentOS
Client Operating System Version 7.7.1908
Client Kernel Version 3.10.0-1062.1.1.el7.x86_64

DATA SERVER

Storage Type NVMe
Volatile Memory 187GB
Storage Interface NVMe over Fabric
Network IB
Software Version 2.13.0
OS Version 7.7.1908

INFORMATION

Client Nodes 59
Client Total Procs 944
Metadata Nodes 4
Metadata Storage Devices 2
Data Nodes 55
Data Storage Devices 109

METADATA

Easy Write 173.96 kIOP/s
Easy Stat 774.97 kIOP/s
Easy Delete 131.04 kIOP/s
Hard Write 87.97 kIOP/s
Hard Read 295.10 kIOP/s
Hard Stat 390.28 kIOP/s
Hard Delete 79.21 kIOP/s

Submitted Files

ior_easy_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 08:27:16 2020
Command line        : /opt/basis/rsc/io500/io500-app/bin/ior -r -R -a POSIX -t 2m --posix.odirect -v -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy/ior_file_easy -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy/stonewall
Machine             : Linux n06p001
Start time skew across all tasks: 983255.50 sec
TestID              : 0
StartTime           : Sat Jul 11 08:27:16 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy
FS                  : 196.3 TiB   Used FS: 23.4%   Inodes: 905.7 Mi   Used Inodes: 19.0%
Participating tasks: 944
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 8930.66 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Sat Jul 11 08:27:16 2020

WARNING: Expected aggregate file size       = 9819368980480000.
WARNING: Stat() of aggregate file size      = 49211668168704.
WARNING: Using actual aggregate bytes moved = 49211668168704.
read      201908     100999     0.000047    10158080000 2048.00    0.063228   232.34     0.088618   232.44     0   
Max Read:  201908.10 MiB/sec (211715.99 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read       201908.10  201908.10  201908.10       0.00  100954.05  100954.05  100954.05       0.00  232.44191         NA            NA     0    944  16    1   1     1        1         0    0      1 10401873920000  2097152 46931904.0 POSIX      0
Finished            : Sat Jul 11 08:31:09 2020
ior_easy_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 07:25:51 2020
Command line        : /opt/basis/rsc/io500/io500-app/bin/ior -w -a POSIX -t 2m --posix.odirect -v -b 9920000m -F -i 1 -C -Q 1 -g -G 27 -k -e -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy/ior_file_easy -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux n06p001
Start time skew across all tasks: 983255.56 sec
TestID              : 0
StartTime           : Sat Jul 11 07:25:51 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy
FS                  : 196.3 TiB   Used FS: 0.1%   Inodes: 793.2 Mi   Used Inodes: 0.0%
Participating tasks: 944
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_easy/ior_file_easy
access              : file-per-process
type                : independent
segments            : 1
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 2 MiB
blocksize           : 9.46 TiB
aggregate filesize  : 8930.66 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Sat Jul 11 07:25:52 2020
271: stonewalling pairs accessed: 20973
266: stonewalling pairs accessed: 20903
821: stonewalling pairs accessed: 21137
478: stonewalling pairs accessed: 23656
96: stonewalling pairs accessed: 20687
496: stonewalling pairs accessed: 24116
479: stonewalling pairs accessed: 23899
501: stonewalling pairs accessed: 23981
362: stonewalling pairs accessed: 20922
887: stonewalling pairs accessed: 21324
893: stonewalling pairs accessed: 21146
363: stonewalling pairs accessed: 23954
361: stonewalling pairs accessed: 21055
366: stonewalling pairs accessed: 23589
823: stonewalling pairs accessed: 21155
71: stonewalling pairs accessed: 21283
880: stonewalling pairs accessed: 24099
66: stonewalling pairs accessed: 21125
105: stonewalling pairs accessed: 21100
499: stonewalling pairs accessed: 23933
98: stonewalling pairs accessed: 21293
342: stonewalling pairs accessed: 21036
345: stonewalling pairs accessed: 21127
943: stonewalling pairs accessed: 23936
247: stonewalling pairs accessed: 20529
351: stonewalling pairs accessed: 21399
242: stonewalling pairs accessed: 21052
338: stonewalling pairs accessed: 21215
237: stonewalling pairs accessed: 21814
892: stonewalling pairs accessed: 20945
349: stonewalling pairs accessed: 21290
217: stonewalling pairs accessed: 23750
25: stonewalling pairs accessed: 23410
248: stonewalling pairs accessed: 21313
75: stonewalling pairs accessed: 23833
24: stonewalling pairs accessed: 20841
914: stonewalling pairs accessed: 23811
387: stonewalling pairs accessed: 21122
753: stonewalling pairs accessed: 23728
393: stonewalling pairs accessed: 23510
16: stonewalling pairs accessed: 20644
529: stonewalling pairs accessed: 24075
922: stonewalling pairs accessed: 23643
855: stonewalling pairs accessed: 20958
81: stonewalling pairs accessed: 20705
246: stonewalling pairs accessed: 20949
864: stonewalling pairs accessed: 20772
541: stonewalling pairs accessed: 21233
68: stonewalling pairs accessed: 21014
533: stonewalling pairs accessed: 21168
862: stonewalling pairs accessed: 23827
538: stonewalling pairs accessed: 23920
220: stonewalling pairs accessed: 23646
5: stonewalling pairs accessed: 23741
634: stonewalling pairs accessed: 24609
398: stonewalling pairs accessed: 20840
716: stonewalling pairs accessed: 20693
631: stonewalling pairs accessed: 21060
240: stonewalling pairs accessed: 21124
583: stonewalling pairs accessed: 20793
239: stonewalling pairs accessed: 20929
147: stonewalling pairs accessed: 20958
863: stonewalling pairs accessed: 21151
900: stonewalling pairs accessed: 21179
532: stonewalling pairs accessed: 23839
622: stonewalling pairs accessed: 23894
89: stonewalling pairs accessed: 23978
638: stonewalling pairs accessed: 21140
704: stonewalling pairs accessed: 24396
643: stonewalling pairs accessed: 21124
715: stonewalling pairs accessed: 21066
80: stonewalling pairs accessed: 20824
767: stonewalling pairs accessed: 21043
924: stonewalling pairs accessed: 23496
869: stonewalling pairs accessed: 20931
630: stonewalling pairs accessed: 23736
9: stonewalling pairs accessed: 23319
917: stonewalling pairs accessed: 20957
626: stonewalling pairs accessed: 21001
872: stonewalling pairs accessed: 20788
43: stonewalling pairs accessed: 23193
899: stonewalling pairs accessed: 24018
644: stonewalling pairs accessed: 24485
528: stonewalling pairs accessed: 21033
592: stonewalling pairs accessed: 23845
637: stonewalling pairs accessed: 20785
47: stonewalling pairs accessed: 23592
545: stonewalling pairs accessed: 23482
755: stonewalling pairs accessed: 20983
911: stonewalling pairs accessed: 20878
721: stonewalling pairs accessed: 21215
253: stonewalling pairs accessed: 20618
608: stonewalling pairs accessed: 23632
916: stonewalling pairs accessed: 20938
594: stonewalling pairs accessed: 21148
19: stonewalling pairs accessed: 20887
2: stonewalling pairs accessed: 20389
536: stonewalling pairs accessed: 21342
557: stonewalling pairs accessed: 20990
93: stonewalling pairs accessed: 20940
559: stonewalling pairs accessed: 20835
556: stonewalling pairs accessed: 20946
180: stonewalling pairs accessed: 20917
754: stonewalling pairs accessed: 23780
596: stonewalling pairs accessed: 21183
182: stonewalling pairs accessed: 20915
794: stonewalling pairs accessed: 20780
497: stonewalling pairs accessed: 21050
787: stonewalling pairs accessed: 24141
837: stonewalling pairs accessed: 20970
639: stonewalling pairs accessed: 21021
179: stonewalling pairs accessed: 20971
566: stonewalling pairs accessed: 23418
758: stonewalling pairs accessed: 21177
790: stonewalling pairs accessed: 20804
709: stonewalling pairs accessed: 21413
717: stonewalling pairs accessed: 21219
890: stonewalling pairs accessed: 21109
599: stonewalling pairs accessed: 21126
648: stonewalling pairs accessed: 21105
273: stonewalling pairs accessed: 21357
874: stonewalling pairs accessed: 23731
285: stonewalling pairs accessed: 23657
727: stonewalling pairs accessed: 23715
553: stonewalling pairs accessed: 24378
128: stonewalling pairs accessed: 21211
95: stonewalling pairs accessed: 21329
912: stonewalling pairs accessed: 21156
934: stonewalling pairs accessed: 21046
90: stonewalling pairs accessed: 21075
277: stonewalling pairs accessed: 20634
156: stonewalling pairs accessed: 23345
42: stonewalling pairs accessed: 23625
442: stonewalling pairs accessed: 21027
510: stonewalling pairs accessed: 20725
94: stonewalling pairs accessed: 20652
707: stonewalling pairs accessed: 21155
433: stonewalling pairs accessed: 20992
508: stonewalling pairs accessed: 21483
621: stonewalling pairs accessed: 21021
148: stonewalling pairs accessed: 21553
3: stonewalling pairs accessed: 20861
734: stonewalling pairs accessed: 20884
834: stonewalling pairs accessed: 24406
795: stonewalling pairs accessed: 20893
435: stonewalling pairs accessed: 21169
587: stonewalling pairs accessed: 23858
842: stonewalling pairs accessed: 21287
155: stonewalling pairs accessed: 23589
37: stonewalling pairs accessed: 23583
714: stonewalling pairs accessed: 24122
555: stonewalling pairs accessed: 21248
879: stonewalling pairs accessed: 20806
221: stonewalling pairs accessed: 23527
845: stonewalling pairs accessed: 24116
561: stonewalling pairs accessed: 21283
935: stonewalling pairs accessed: 20792
23: stonewalling pairs accessed: 20902
44: stonewalling pairs accessed: 23482
159: stonewalling pairs accessed: 21224
791: stonewalling pairs accessed: 20453
571: stonewalling pairs accessed: 21072
368: stonewalling pairs accessed: 20997
279: stonewalling pairs accessed: 20933
581: stonewalling pairs accessed: 23432
833: stonewalling pairs accessed: 20966
584: stonewalling pairs accessed: 23797
238: stonewalling pairs accessed: 21029
150: stonewalling pairs accessed: 21273
4: stonewalling pairs accessed: 20789
509: stonewalling pairs accessed: 20857
223: stonewalling pairs accessed: 20934
723: stonewalling pairs accessed: 23779
591: stonewalling pairs accessed: 21230
141: stonewalling pairs accessed: 21014
798: stonewalling pairs accessed: 20946
370: stonewalling pairs accessed: 21041
923: stonewalling pairs accessed: 21077
796: stonewalling pairs accessed: 21093
353: stonewalling pairs accessed: 20984
856: stonewalling pairs accessed: 21183
13: stonewalling pairs accessed: 20648
15: stonewalling pairs accessed: 20675
567: stonewalling pairs accessed: 21187
752: stonewalling pairs accessed: 21042
274: stonewalling pairs accessed: 21167
282: stonewalling pairs accessed: 23654
26: stonewalling pairs accessed: 23601
374: stonewalling pairs accessed: 23363
615: stonewalling pairs accessed: 23753
278: stonewalling pairs accessed: 24441
396: stonewalling pairs accessed: 23065
670: stonewalling pairs accessed: 23472
799: stonewalling pairs accessed: 23545
276: stonewalling pairs accessed: 20664
364: stonewalling pairs accessed: 23537
41: stonewalling pairs accessed: 23861
735: stonewalling pairs accessed: 24079
797: stonewalling pairs accessed: 23431
611: stonewalling pairs accessed: 23796
841: stonewalling pairs accessed: 21296
793: stonewalling pairs accessed: 21137
871: stonewalling pairs accessed: 23433
792: stonewalling pairs accessed: 23850
540: stonewalling pairs accessed: 23923
542: stonewalling pairs accessed: 23485
21: stonewalling pairs accessed: 21111
819: stonewalling pairs accessed: 21418
430: stonewalling pairs accessed: 21009
211: stonewalling pairs accessed: 21055
146: stonewalling pairs accessed: 21094
784: stonewalling pairs accessed: 23863
729: stonewalling pairs accessed: 24090
847: stonewalling pairs accessed: 24357
137: stonewalling pairs accessed: 23983
904: stonewalling pairs accessed: 23670
582: stonewalling pairs accessed: 24004
382: stonewalling pairs accessed: 24148
830: stonewalling pairs accessed: 23720
378: stonewalling pairs accessed: 21287
881: stonewalling pairs accessed: 20875
756: stonewalling pairs accessed: 23886
839: stonewalling pairs accessed: 21220
526: stonewalling pairs accessed: 23544
726: stonewalling pairs accessed: 21121
375: stonewalling pairs accessed: 21111
144: stonewalling pairs accessed: 20782
140: stonewalling pairs accessed: 21257
785: stonewalling pairs accessed: 20834
153: stonewalling pairs accessed: 23805
379: stonewalling pairs accessed: 20577
109: stonewalling pairs accessed: 23322
873: stonewalling pairs accessed: 20973
380: stonewalling pairs accessed: 21173
198: stonewalling pairs accessed: 20746
588: stonewalling pairs accessed: 20641
590: stonewalling pairs accessed: 20945
665: stonewalling pairs accessed: 21275
568: stonewalling pairs accessed: 21397
569: stonewalling pairs accessed: 21077
372: stonewalling pairs accessed: 21003
595: stonewalling pairs accessed: 20618
272: stonewalling pairs accessed: 20950
809: stonewalling pairs accessed: 21040
38: stonewalling pairs accessed: 23678
713: stonewalling pairs accessed: 20762
825: stonewalling pairs accessed: 21282
574: stonewalling pairs accessed: 20950
657: stonewalling pairs accessed: 21263
169: stonewalling pairs accessed: 23745
789: stonewalling pairs accessed: 20805
606: stonewalling pairs accessed: 21115
201: stonewalling pairs accessed: 21342
564: stonewalling pairs accessed: 21166
725: stonewalling pairs accessed: 21249
570: stonewalling pairs accessed: 20985
283: stonewalling pairs accessed: 23523
136: stonewalling pairs accessed: 24163
280: stonewalling pairs accessed: 23182
158: stonewalling pairs accessed: 23628
162: stonewalling pairs accessed: 23516
390: stonewalling pairs accessed: 24259
469: stonewalling pairs accessed: 21134
535: stonewalling pairs accessed: 21335
76: stonewalling pairs accessed: 20982
208: stonewalling pairs accessed: 21142
287: stonewalling pairs accessed: 20842
578: stonewalling pairs accessed: 23398
660: stonewalling pairs accessed: 23706
391: stonewalling pairs accessed: 20779
562: stonewalling pairs accessed: 21304
213: stonewalling pairs accessed: 20603
655: stonewalling pairs accessed: 21254
218: stonewalling pairs accessed: 21364
222: stonewalling pairs accessed: 20686
209: stonewalling pairs accessed: 23817
910: stonewalling pairs accessed: 21232
826: stonewalling pairs accessed: 21380
907: stonewalling pairs accessed: 20533
480: stonewalling pairs accessed: 21144
938: stonewalling pairs accessed: 21092
786: stonewalling pairs accessed: 23602
184: stonewalling pairs accessed: 20637
788: stonewalling pairs accessed: 21134
202: stonewalling pairs accessed: 20820
930: stonewalling pairs accessed: 20823
710: stonewalling pairs accessed: 24391
216: stonewalling pairs accessed: 23878
805: stonewalling pairs accessed: 20935
204: stonewalling pairs accessed: 21336
205: stonewalling pairs accessed: 24712
77: stonewalling pairs accessed: 20741
195: stonewalling pairs accessed: 23929
523: stonewalling pairs accessed: 20576
154: stonewalling pairs accessed: 23849
882: stonewalling pairs accessed: 20567
858: stonewalling pairs accessed: 20471
300: stonewalling pairs accessed: 20839
199: stonewalling pairs accessed: 23743
176: stonewalling pairs accessed: 20715
728: stonewalling pairs accessed: 21251
730: stonewalling pairs accessed: 21365
563: stonewalling pairs accessed: 20753
429: stonewalling pairs accessed: 23932
290: stonewalling pairs accessed: 24082
215: stonewalling pairs accessed: 21365
371: stonewalling pairs accessed: 23835
281: stonewalling pairs accessed: 23932
849: stonewalling pairs accessed: 23761
440: stonewalling pairs accessed: 20752
470: stonewalling pairs accessed: 23607
688: stonewalling pairs accessed: 20992
724: stonewalling pairs accessed: 23504
377: stonewalling pairs accessed: 23060
818: stonewalling pairs accessed: 21041
597: stonewalling pairs accessed: 23513
781: stonewalling pairs accessed: 23564
286: stonewalling pairs accessed: 20983
695: stonewalling pairs accessed: 21204
661: stonewalling pairs accessed: 21287
472: stonewalling pairs accessed: 21131
207: stonewalling pairs accessed: 21682
722: stonewalling pairs accessed: 21243
577: stonewalling pairs accessed: 21588
134: stonewalling pairs accessed: 21068
138: stonewalling pairs accessed: 23532
139: stonewalling pairs accessed: 21247
846: stonewalling pairs accessed: 21352
857: stonewalling pairs accessed: 21161
373: stonewalling pairs accessed: 23865
187: stonewalling pairs accessed: 21137
403: stonewalling pairs accessed: 23173
432: stonewalling pairs accessed: 23430
389: stonewalling pairs accessed: 21669
415: stonewalling pairs accessed: 20821
732: stonewalling pairs accessed: 23715
813: stonewalling pairs accessed: 20666
905: stonewalling pairs accessed: 21082
720: stonewalling pairs accessed: 22696
589: stonewalling pairs accessed: 21044
142: stonewalling pairs accessed: 21084
838: stonewalling pairs accessed: 21313
560: stonewalling pairs accessed: 23667
851: stonewalling pairs accessed: 24644
598: stonewalling pairs accessed: 23763
130: stonewalling pairs accessed: 21469
102: stonewalling pairs accessed: 23496
65: stonewalling pairs accessed: 20786
369: stonewalling pairs accessed: 21202
408: stonewalling pairs accessed: 21219
843: stonewalling pairs accessed: 21146
376: stonewalling pairs accessed: 20963
565: stonewalling pairs accessed: 24333
942: stonewalling pairs accessed: 20552
898: stonewalling pairs accessed: 23979
418: stonewalling pairs accessed: 20886
203: stonewalling pairs accessed: 23849
719: stonewalling pairs accessed: 24051
601: stonewalling pairs accessed: 20618
482: stonewalling pairs accessed: 21111
405: stonewalling pairs accessed: 24473
768: stonewalling pairs accessed: 20510
196: stonewalling pairs accessed: 21229
427: stonewalling pairs accessed: 20564
465: stonewalling pairs accessed: 20897
275: stonewalling pairs accessed: 20733
579: stonewalling pairs accessed: 20705
383: stonewalling pairs accessed: 20806
612: stonewalling pairs accessed: 21039
614: stonewalling pairs accessed: 20786
602: stonewalling pairs accessed: 20891
850: stonewalling pairs accessed: 21037
852: stonewalling pairs accessed: 20927
861: stonewalling pairs accessed: 23756
658: stonewalling pairs accessed: 20964
301: stonewalling pairs accessed: 21600
477: stonewalling pairs accessed: 21380
425: stonewalling pairs accessed: 21170
149: stonewalling pairs accessed: 20952
178: stonewalling pairs accessed: 20819
131: stonewalling pairs accessed: 20700
132: stonewalling pairs accessed: 21411
133: stonewalling pairs accessed: 24722
181: stonewalling pairs accessed: 20642
107: stonewalling pairs accessed: 20712
733: stonewalling pairs accessed: 20904
585: stonewalling pairs accessed: 21057
298: stonewalling pairs accessed: 20970
315: stonewalling pairs accessed: 20600
593: stonewalling pairs accessed: 21114
656: stonewalling pairs accessed: 20981
539: stonewalling pairs accessed: 21341
603: stonewalling pairs accessed: 21168
575: stonewalling pairs accessed: 23667
770: stonewalling pairs accessed: 21437
414: stonewalling pairs accessed: 21230
779: stonewalling pairs accessed: 20925
698: stonewalling pairs accessed: 20645
111: stonewalling pairs accessed: 21153
58: stonewalling pairs accessed: 20951
906: stonewalling pairs accessed: 20918
145: stonewalling pairs accessed: 24091
193: stonewalling pairs accessed: 20893
486: stonewalling pairs accessed: 24135
426: stonewalling pairs accessed: 20978
57: stonewalling pairs accessed: 23412
534: stonewalling pairs accessed: 23611
516: stonewalling pairs accessed: 21023
662: stonewalling pairs accessed: 23999
731: stonewalling pairs accessed: 20904
543: stonewalling pairs accessed: 20787
412: stonewalling pairs accessed: 21051
143: stonewalling pairs accessed: 21191
135: stonewalling pairs accessed: 22739
927: stonewalling pairs accessed: 21307
586: stonewalling pairs accessed: 23763
443: stonewalling pairs accessed: 20840
520: stonewalling pairs accessed: 23686
192: stonewalling pairs accessed: 21229
632: stonewalling pairs accessed: 23473
183: stonewalling pairs accessed: 20954
773: stonewalling pairs accessed: 20913
806: stonewalling pairs accessed: 21096
50: stonewalling pairs accessed: 21096
554: stonewalling pairs accessed: 21061
194: stonewalling pairs accessed: 24163
191: stonewalling pairs accessed: 21050
777: stonewalling pairs accessed: 20843
832: stonewalling pairs accessed: 21108
531: stonewalling pairs accessed: 21305
200: stonewalling pairs accessed: 23621
835: stonewalling pairs accessed: 21134
932: stonewalling pairs accessed: 21145
219: stonewalling pairs accessed: 21420
836: stonewalling pairs accessed: 23956
189: stonewalling pairs accessed: 20602
840: stonewalling pairs accessed: 20926
186: stonewalling pairs accessed: 21142
859: stonewalling pairs accessed: 23559
860: stonewalling pairs accessed: 21165
409: stonewalling pairs accessed: 21226
420: stonewalling pairs accessed: 21637
580: stonewalling pairs accessed: 21267
718: stonewalling pairs accessed: 21103
91: stonewalling pairs accessed: 20916
416: stonewalling pairs accessed: 23660
413: stonewalling pairs accessed: 21060
708: stonewalling pairs accessed: 21214
576: stonewalling pairs accessed: 20948
173: stonewalling pairs accessed: 20873
48: stonewalling pairs accessed: 23136
844: stonewalling pairs accessed: 21588
306: stonewalling pairs accessed: 21354
776: stonewalling pairs accessed: 23730
705: stonewalling pairs accessed: 21264
667: stonewalling pairs accessed: 23587
646: stonewalling pairs accessed: 21147
663: stonewalling pairs accessed: 23779
774: stonewalling pairs accessed: 23889
848: stonewalling pairs accessed: 23712
52: stonewalling pairs accessed: 21115
210: stonewalling pairs accessed: 20939
411: stonewalling pairs accessed: 21649
212: stonewalling pairs accessed: 23683
572: stonewalling pairs accessed: 21405
692: stonewalling pairs accessed: 23407
295: stonewalling pairs accessed: 21064
313: stonewalling pairs accessed: 21010
299: stonewalling pairs accessed: 21079
771: stonewalling pairs accessed: 21254
446: stonewalling pairs accessed: 20985
901: stonewalling pairs accessed: 23474
284: stonewalling pairs accessed: 20859
605: stonewalling pairs accessed: 20462
152: stonewalling pairs accessed: 21007
641: stonewalling pairs accessed: 20972
757: stonewalling pairs accessed: 24238
853: stonewalling pairs accessed: 20542
783: stonewalling pairs accessed: 21018
782: stonewalling pairs accessed: 23543
292: stonewalling pairs accessed: 21352
404: stonewalling pairs accessed: 21231
434: stonewalling pairs accessed: 20982
854: stonewalling pairs accessed: 20878
659: stonewalling pairs accessed: 21006
666: stonewalling pairs accessed: 23671
59: stonewalling pairs accessed: 23387
491: stonewalling pairs accessed: 23651
62: stonewalling pairs accessed: 20601
304: stonewalling pairs accessed: 21279
60: stonewalling pairs accessed: 23562
336: stonewalling pairs accessed: 21081
483: stonewalling pairs accessed: 20822
870: stonewalling pairs accessed: 21312
314: stonewalling pairs accessed: 20977
129: stonewalling pairs accessed: 23627
407: stonewalling pairs accessed: 23608
419: stonewalling pairs accessed: 21331
206: stonewalling pairs accessed: 23916
438: stonewalling pairs accessed: 23991
780: stonewalling pairs accessed: 20814
311: stonewalling pairs accessed: 23376
764: stonewalling pairs accessed: 23859
664: stonewalling pairs accessed: 21002
504: stonewalling pairs accessed: 20836
190: stonewalling pairs accessed: 20974
492: stonewalling pairs accessed: 23643
689: stonewalling pairs accessed: 20921
711: stonewalling pairs accessed: 24015
381: stonewalling pairs accessed: 23829
214: stonewalling pairs accessed: 20518
640: stonewalling pairs accessed: 24296
654: stonewalling pairs accessed: 20940
401: stonewalling pairs accessed: 23885
112: stonewalling pairs accessed: 21401
305: stonewalling pairs accessed: 21542
185: stonewalling pairs accessed: 20894
34: stonewalling pairs accessed: 23502
444: stonewalling pairs accessed: 21084
522: stonewalling pairs accessed: 22145
642: stonewalling pairs accessed: 20845
51: stonewalling pairs accessed: 20654
385: stonewalling pairs accessed: 21493
397: stonewalling pairs accessed: 20826
706: stonewalling pairs accessed: 21384
447: stonewalling pairs accessed: 23452
439: stonewalling pairs accessed: 20693
488: stonewalling pairs accessed: 23721
537: stonewalling pairs accessed: 23837
485: stonewalling pairs accessed: 21106
878: stonewalling pairs accessed: 21018
671: stonewalling pairs accessed: 21312
921: stonewalling pairs accessed: 24056
326: stonewalling pairs accessed: 21216
230: stonewalling pairs accessed: 24026
151: stonewalling pairs accessed: 21269
30: stonewalling pairs accessed: 23752
31: stonewalling pairs accessed: 20711
778: stonewalling pairs accessed: 20998
573: stonewalling pairs accessed: 23838
406: stonewalling pairs accessed: 20966
177: stonewalling pairs accessed: 21297
604: stonewalling pairs accessed: 23849
312: stonewalling pairs accessed: 21330
607: stonewalling pairs accessed: 20808
903: stonewalling pairs accessed: 20975
668: stonewalling pairs accessed: 20933
691: stonewalling pairs accessed: 21035
651: stonewalling pairs accessed: 21108
653: stonewalling pairs accessed: 23744
157: stonewalling pairs accessed: 20983
652: stonewalling pairs accessed: 21456
17: stonewalling pairs accessed: 23550
423: stonewalling pairs accessed: 23456
428: stonewalling pairs accessed: 23938
441: stonewalling pairs accessed: 21117
445: stonewalling pairs accessed: 23898
55: stonewalling pairs accessed: 20654
63: stonewalling pairs accessed: 20786
759: stonewalling pairs accessed: 21243
760: stonewalling pairs accessed: 21430
166: stonewalling pairs accessed: 20662
702: stonewalling pairs accessed: 23432
489: stonewalling pairs accessed: 20869
493: stonewalling pairs accessed: 20882
647: stonewalling pairs accessed: 24064
507: stonewalling pairs accessed: 21173
803: stonewalling pairs accessed: 21300
487: stonewalling pairs accessed: 21003
481: stonewalling pairs accessed: 21551
941: stonewalling pairs accessed: 21353
699: stonewalling pairs accessed: 23557
28: stonewalling pairs accessed: 20767
54: stonewalling pairs accessed: 23305
600: stonewalling pairs accessed: 23663
307: stonewalling pairs accessed: 21080
422: stonewalling pairs accessed: 21174
316: stonewalling pairs accessed: 20614
468: stonewalling pairs accessed: 20811
865: stonewalling pairs accessed: 24028
743: stonewalling pairs accessed: 20901
319: stonewalling pairs accessed: 20927
354: stonewalling pairs accessed: 21244
188: stonewalling pairs accessed: 20937
762: stonewalling pairs accessed: 21101
804: stonewalling pairs accessed: 21008
49: stonewalling pairs accessed: 21082
530: stonewalling pairs accessed: 21141
875: stonewalling pairs accessed: 21295
288: stonewalling pairs accessed: 21129
46: stonewalling pairs accessed: 20827
484: stonewalling pairs accessed: 24055
512: stonewalling pairs accessed: 21307
931: stonewalling pairs accessed: 21169
82: stonewalling pairs accessed: 20791
712: stonewalling pairs accessed: 21016
318: stonewalling pairs accessed: 21022
163: stonewalling pairs accessed: 20597
513: stonewalling pairs accessed: 23961
74: stonewalling pairs accessed: 21047
519: stonewalling pairs accessed: 24463
84: stonewalling pairs accessed: 20921
548: stonewalling pairs accessed: 20627
437: stonewalling pairs accessed: 20775
514: stonewalling pairs accessed: 21313
515: stonewalling pairs accessed: 23542
197: stonewalling pairs accessed: 22586
736: stonewalling pairs accessed: 20955
317: stonewalling pairs accessed: 23740
616: stonewalling pairs accessed: 21110
322: stonewalling pairs accessed: 21257
172: stonewalling pairs accessed: 20880
701: stonewalling pairs accessed: 20658
765: stonewalling pairs accessed: 21057
742: stonewalling pairs accessed: 20786
650: stonewalling pairs accessed: 21104
751: stonewalling pairs accessed: 20815
700: stonewalling pairs accessed: 21173
645: stonewalling pairs accessed: 24089
116: stonewalling pairs accessed: 21020
740: stonewalling pairs accessed: 21174
703: stonewalling pairs accessed: 20578
473: stonewalling pairs accessed: 20511
87: stonewalling pairs accessed: 23549
810: stonewalling pairs accessed: 23646
392: stonewalling pairs accessed: 20599
741: stonewalling pairs accessed: 20757
119: stonewalling pairs accessed: 20958
53: stonewalling pairs accessed: 20968
365: stonewalling pairs accessed: 24108
500: stonewalling pairs accessed: 21026
511: stonewalling pairs accessed: 21697
436: stonewalling pairs accessed: 23399
121: stonewalling pairs accessed: 21109
761: stonewalling pairs accessed: 23674
802: stonewalling pairs accessed: 20884
649: stonewalling pairs accessed: 23674
814: stonewalling pairs accessed: 21255
125: stonewalling pairs accessed: 21142
124: stonewalling pairs accessed: 20998
748: stonewalling pairs accessed: 23603
490: stonewalling pairs accessed: 23494
495: stonewalling pairs accessed: 21008
118: stonewalling pairs accessed: 22348
232: stonewalling pairs accessed: 21366
524: stonewalling pairs accessed: 21086
625: stonewalling pairs accessed: 24196
815: stonewalling pairs accessed: 24132
32: stonewalling pairs accessed: 23317
174: stonewalling pairs accessed: 20898
808: stonewalling pairs accessed: 20920
175: stonewalling pairs accessed: 21103
766: stonewalling pairs accessed: 24142
876: stonewalling pairs accessed: 23517
877: stonewalling pairs accessed: 23387
386: stonewalling pairs accessed: 23568
633: stonewalling pairs accessed: 21190
85: stonewalling pairs accessed: 23479
457: stonewalling pairs accessed: 20561
884: stonewalling pairs accessed: 21376
889: stonewalling pairs accessed: 24314
384: stonewalling pairs accessed: 23416
56: stonewalling pairs accessed: 23175
800: stonewalling pairs accessed: 20984
431: stonewalling pairs accessed: 23656
460: stonewalling pairs accessed: 20869
521: stonewalling pairs accessed: 21094
915: stonewalling pairs accessed: 24214
123: stonewalling pairs accessed: 21132
235: stonewalling pairs accessed: 23312
627: stonewalling pairs accessed: 20941
333: stonewalling pairs accessed: 21374
696: stonewalling pairs accessed: 21086
775: stonewalling pairs accessed: 23661
61: stonewalling pairs accessed: 21155
448: stonewalling pairs accessed: 20930
918: stonewalling pairs accessed: 21339
636: stonewalling pairs accessed: 21217
772: stonewalling pairs accessed: 23881
36: stonewalling pairs accessed: 23707
610: stonewalling pairs accessed: 21151
807: stonewalling pairs accessed: 21291
122: stonewalling pairs accessed: 20996
115: stonewalling pairs accessed: 21431
518: stonewalling pairs accessed: 21028
494: stonewalling pairs accessed: 23894
867: stonewalling pairs accessed: 23593
738: stonewalling pairs accessed: 21366
609: stonewalling pairs accessed: 21358
161: stonewalling pairs accessed: 23501
694: stonewalling pairs accessed: 23712
801: stonewalling pairs accessed: 20862
886: stonewalling pairs accessed: 24451
739: stonewalling pairs accessed: 20974
164: stonewalling pairs accessed: 21202
669: stonewalling pairs accessed: 23654
746: stonewalling pairs accessed: 20887
168: stonewalling pairs accessed: 21302
678: stonewalling pairs accessed: 23893
749: stonewalling pairs accessed: 20974
88: stonewalling pairs accessed: 21289
358: stonewalling pairs accessed: 24327
812: stonewalling pairs accessed: 21175
745: stonewalling pairs accessed: 21083
340: stonewalling pairs accessed: 23663
929: stonewalling pairs accessed: 21106
750: stonewalling pairs accessed: 21250
628: stonewalling pairs accessed: 20767
744: stonewalling pairs accessed: 20952
498: stonewalling pairs accessed: 20875
459: stonewalling pairs accessed: 20845
525: stonewalling pairs accessed: 21107
400: stonewalling pairs accessed: 21186
462: stonewalling pairs accessed: 20644
763: stonewalling pairs accessed: 20716
388: stonewalling pairs accessed: 23870
394: stonewalling pairs accessed: 21181
395: stonewalling pairs accessed: 23505
456: stonewalling pairs accessed: 21011
92: stonewalling pairs accessed: 23517
868: stonewalling pairs accessed: 20938
113: stonewalling pairs accessed: 21057
64: stonewalling pairs accessed: 20722
78: stonewalling pairs accessed: 23406
86: stonewalling pairs accessed: 23605
127: stonewalling pairs accessed: 21198
117: stonewalling pairs accessed: 20944
685: stonewalling pairs accessed: 20995
620: stonewalling pairs accessed: 21161
347: stonewalling pairs accessed: 20853
421: stonewalling pairs accessed: 20809
399: stonewalling pairs accessed: 21021
359: stonewalling pairs accessed: 20823
902: stonewalling pairs accessed: 20986
908: stonewalling pairs accessed: 21354
20: stonewalling pairs accessed: 20878
22: stonewalling pairs accessed: 20583
527: stonewalling pairs accessed: 21017
45: stonewalling pairs accessed: 20966
811: stonewalling pairs accessed: 20884
114: stonewalling pairs accessed: 20817
677: stonewalling pairs accessed: 24049
939: stonewalling pairs accessed: 22336
126: stonewalling pairs accessed: 21056
679: stonewalling pairs accessed: 21162
684: stonewalling pairs accessed: 23610
920: stonewalling pairs accessed: 20908
909: stonewalling pairs accessed: 21012
503: stonewalling pairs accessed: 20956
310: stonewalling pairs accessed: 23486
424: stonewalling pairs accessed: 24199
449: stonewalling pairs accessed: 21314
629: stonewalling pairs accessed: 23397
505: stonewalling pairs accessed: 20800
517: stonewalling pairs accessed: 21118
919: stonewalling pairs accessed: 20929
79: stonewalling pairs accessed: 20612
866: stonewalling pairs accessed: 21098
227: stonewalling pairs accessed: 23890
635: stonewalling pairs accessed: 20867
165: stonewalling pairs accessed: 20611
674: stonewalling pairs accessed: 23635
18: stonewalling pairs accessed: 23478
502: stonewalling pairs accessed: 23833
891: stonewalling pairs accessed: 21107
623: stonewalling pairs accessed: 23731
170: stonewalling pairs accessed: 23393
680: stonewalling pairs accessed: 21127
463: stonewalling pairs accessed: 20978
171: stonewalling pairs accessed: 21314
451: stonewalling pairs accessed: 21214
321: stonewalling pairs accessed: 20979
675: stonewalling pairs accessed: 21452
687: stonewalling pairs accessed: 21124
327: stonewalling pairs accessed: 20752
461: stonewalling pairs accessed: 22155
897: stonewalling pairs accessed: 21083
697: stonewalling pairs accessed: 23303
33: stonewalling pairs accessed: 21079
228: stonewalling pairs accessed: 23585
888: stonewalling pairs accessed: 20936
35: stonewalling pairs accessed: 23660
236: stonewalling pairs accessed: 20944
506: stonewalling pairs accessed: 20957
926: stonewalling pairs accessed: 23976
297: stonewalling pairs accessed: 24858
450: stonewalling pairs accessed: 20826
624: stonewalling pairs accessed: 24189
224: stonewalling pairs accessed: 21462
617: stonewalling pairs accessed: 23575
913: stonewalling pairs accessed: 23760
940: stonewalling pairs accessed: 21376
676: stonewalling pairs accessed: 21175
683: stonewalling pairs accessed: 21109
619: stonewalling pairs accessed: 20766
452: stonewalling pairs accessed: 20926
29: stonewalling pairs accessed: 23210
328: stonewalling pairs accessed: 21164
332: stonewalling pairs accessed: 21203
308: stonewalling pairs accessed: 24036
120: stonewalling pairs accessed: 21198
417: stonewalling pairs accessed: 20978
83: stonewalling pairs accessed: 23683
39: stonewalling pairs accessed: 23634
925: stonewalling pairs accessed: 23609
894: stonewalling pairs accessed: 23978
618: stonewalling pairs accessed: 21228
690: stonewalling pairs accessed: 23528
769: stonewalling pairs accessed: 23744
40: stonewalling pairs accessed: 23305
936: stonewalling pairs accessed: 20876
27: stonewalling pairs accessed: 20996
937: stonewalling pairs accessed: 21328
613: stonewalling pairs accessed: 21463
320: stonewalling pairs accessed: 21032
334: stonewalling pairs accessed: 21032
234: stonewalling pairs accessed: 23971
335: stonewalling pairs accessed: 21283
824: stonewalling pairs accessed: 21139
226: stonewalling pairs accessed: 21433
225: stonewalling pairs accessed: 20887
296: stonewalling pairs accessed: 21495
330: stonewalling pairs accessed: 21278
883: stonewalling pairs accessed: 21227
453: stonewalling pairs accessed: 20992
167: stonewalling pairs accessed: 21113
402: stonewalling pairs accessed: 23722
928: stonewalling pairs accessed: 20999
339: stonewalling pairs accessed: 23736
672: stonewalling pairs accessed: 21237
829: stonewalling pairs accessed: 24218
546: stonewalling pairs accessed: 21111
885: stonewalling pairs accessed: 20799
231: stonewalling pairs accessed: 23330
309: stonewalling pairs accessed: 21316
325: stonewalling pairs accessed: 20834
550: stonewalling pairs accessed: 23342
293: stonewalling pairs accessed: 23520
341: stonewalling pairs accessed: 23372
294: stonewalling pairs accessed: 20746
324: stonewalling pairs accessed: 21483
302: stonewalling pairs accessed: 20701
303: stonewalling pairs accessed: 20866
291: stonewalling pairs accessed: 20960
233: stonewalling pairs accessed: 23917
160: stonewalling pairs accessed: 23201
896: stonewalling pairs accessed: 23615
343: stonewalling pairs accessed: 21008
410: stonewalling pairs accessed: 23733
455: stonewalling pairs accessed: 20952
933: stonewalling pairs accessed: 21128
289: stonewalling pairs accessed: 20915
346: stonewalling pairs accessed: 23725
693: stonewalling pairs accessed: 23680
1: stonewalling pairs accessed: 20572
737: stonewalling pairs accessed: 20907
100: stonewalling pairs accessed: 21259
820: stonewalling pairs accessed: 21071
831: stonewalling pairs accessed: 21059
67: stonewalling pairs accessed: 21208
895: stonewalling pairs accessed: 20893
229: stonewalling pairs accessed: 24510
681: stonewalling pairs accessed: 24123
331: stonewalling pairs accessed: 21302
12: stonewalling pairs accessed: 21210
817: stonewalling pairs accessed: 21419
828: stonewalling pairs accessed: 21031
816: stonewalling pairs accessed: 20964
544: stonewalling pairs accessed: 20915
73: stonewalling pairs accessed: 20984
822: stonewalling pairs accessed: 24243
108: stonewalling pairs accessed: 21053
355: stonewalling pairs accessed: 21181
10: stonewalling pairs accessed: 20582
549: stonewalling pairs accessed: 21324
673: stonewalling pairs accessed: 24167
7: stonewalling pairs accessed: 23564
558: stonewalling pairs accessed: 21092
827: stonewalling pairs accessed: 21066
352: stonewalling pairs accessed: 23930
323: stonewalling pairs accessed: 21078
329: stonewalling pairs accessed: 21003
356: stonewalling pairs accessed: 21405
110: stonewalling pairs accessed: 20948
69: stonewalling pairs accessed: 21097
99: stonewalling pairs accessed: 20847
747: stonewalling pairs accessed: 23553
458: stonewalling pairs accessed: 20438
70: stonewalling pairs accessed: 21124
72: stonewalling pairs accessed: 23438
6: stonewalling pairs accessed: 23869
14: stonewalling pairs accessed: 23597
11: stonewalling pairs accessed: 20809
97: stonewalling pairs accessed: 20685
551: stonewalling pairs accessed: 23762
360: stonewalling pairs accessed: 23678
682: stonewalling pairs accessed: 23576
104: stonewalling pairs accessed: 21320
547: stonewalling pairs accessed: 24001
0: stonewalling pairs accessed: 20591
103: stonewalling pairs accessed: 21463
357: stonewalling pairs accessed: 21218
552: stonewalling pairs accessed: 20848
367: stonewalling pairs accessed: 21011
101: stonewalling pairs accessed: 20721
106: stonewalling pairs accessed: 20847
344: stonewalling pairs accessed: 21468
454: stonewalling pairs accessed: 21343
686: stonewalling pairs accessed: 23941
337: stonewalling pairs accessed: 21058
8: stonewalling pairs accessed: 23615
350: stonewalling pairs accessed: 23624
464: stonewalling pairs accessed: 23678
348: stonewalling pairs accessed: 20954
467: stonewalling pairs accessed: 21366
471: stonewalling pairs accessed: 21187
466: stonewalling pairs accessed: 21052
476: stonewalling pairs accessed: 20941
251: stonewalling pairs accessed: 21316
255: stonewalling pairs accessed: 20622
475: stonewalling pairs accessed: 20930
474: stonewalling pairs accessed: 21090
241: stonewalling pairs accessed: 20756
244: stonewalling pairs accessed: 20959
245: stonewalling pairs accessed: 24024
252: stonewalling pairs accessed: 21146
243: stonewalling pairs accessed: 23580
250: stonewalling pairs accessed: 20903
249: stonewalling pairs accessed: 20930
254: stonewalling pairs accessed: 22248
261: stonewalling pairs accessed: 21238
267: stonewalling pairs accessed: 20963
256: stonewalling pairs accessed: 21068
264: stonewalling pairs accessed: 21148
259: stonewalling pairs accessed: 20634
258: stonewalling pairs accessed: 20758
268: stonewalling pairs accessed: 20970
262: stonewalling pairs accessed: 20670
265: stonewalling pairs accessed: 20850
270: stonewalling pairs accessed: 20947
269: stonewalling pairs accessed: 20636
263: stonewalling pairs accessed: 20917
257: stonewalling pairs accessed: 21182
260: stonewalling pairs accessed: 21679
stonewalling pairs accessed min: 20389 max: 24858 -- min data: 39.8 GiB mean data: 42.8 GiB time: 300.2s
WARNING: Expected aggregate file size       = 9819368980480000.
WARNING: Stat() of aggregate file size      = 49211668168704.
WARNING: Using actual aggregate bytes moved = 49211668168704.
WARNING: maybe caused by deadlineForStonewalling
write     129139     64602      0.000073    10158080000 2048.00    0.078092   363.24     0.096379   363.42     0   
Max Write: 129138.84 MiB/sec (135411.89 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write      129138.84  129138.84  129138.84       0.00   64569.42   64569.42   64569.42       0.00  363.42206     300.19     137740.48     0    944  16    1   1     1        1         0    0      1 10401873920000  2097152 46931904.0 POSIX      0
Finished            : Sat Jul 11 07:31:55 2020
ior_hard_read
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 08:33:48 2020
Command line        : /opt/basis/rsc/io500/io500-app/bin/ior -r -R -s 34000 -a MPIIO --posix.odirect -v -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard/IOR_file -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard/stonewall
Machine             : Linux n06p001
Start time skew across all tasks: 983257.03 sec
TestID              : 0
StartTime           : Sat Jul 11 08:33:48 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard
FS                  : 196.3 TiB   Used FS: 23.4%   Inodes: 905.7 Mi   Used Inodes: 19.0%
Participating tasks: 944
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 34000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 1.37 TiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing read performance test: Sat Jul 11 08:34:06 2020

WARNING: Expected aggregate file size       = 1508768768000.
WARNING: Stat() of aggregate file size      = 1083162848768.
WARNING: Using actual aggregate bytes moved = 1083162848768.
read      2855.28    66865      344.47      45.91      45.91      17.10      344.61     0.146084   361.78     0   
Max Read:  2855.28 MiB/sec (2993.97 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
read         2855.28    2855.28    2855.28       0.00   63690.73   63690.73   63690.73       0.00  361.78099         NA            NA     0    944  16    1   0     1        1         0    0  34000    47008    47008 1032984.6 MPIIO      0
Finished            : Sat Jul 11 08:39:51 2020
ior_hard_write
IOR-3.3.0+dev: MPI Coordinated Test of Parallel I/O
Began               : Sat Jul 11 07:45:55 2020
Command line        : /opt/basis/rsc/io500/io500-app/bin/ior -w -s 34000 -a MPIIO --posix.odirect -v -i 1 -C -Q 1 -g -G 27 -k -e -t 47008 -b 47008 -o /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard/IOR_file -O stoneWallingStatusFile=/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard/stonewall -O stoneWallingWearOut=1 -D 300
Machine             : Linux n06p001
Start time skew across all tasks: 983255.36 sec
TestID              : 0
StartTime           : Sat Jul 11 07:45:55 2020
Path                : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard
FS                  : 196.3 TiB   Used FS: 22.8%   Inodes: 905.9 Mi   Used Inodes: 12.4%
Participating tasks: 944
Using reorderTasks '-C' (useful to avoid read cache in client)

Options: 
api                 : MPIIO
apiVersion          : (3.1)
test filename       : /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/ior_hard/IOR_file
access              : single-shared-file
type                : independent
segments            : 34000
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 59
tasks               : 944
clients per node    : 16
repetitions         : 1
xfersize            : 47008 bytes
blocksize           : 47008 bytes
aggregate filesize  : 1.37 TiB
stonewallingTime    : 300
stoneWallingWearOut : 1

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
Commencing write performance test: Sat Jul 11 07:46:12 2020
696: stonewalling pairs accessed: 19869
497: stonewalling pairs accessed: 19938
498: stonewalling pairs accessed: 19451
697: stonewalling pairs accessed: 20260
23: stonewalling pairs accessed: 20071
446: stonewalling pairs accessed: 19633
7: stonewalling pairs accessed: 15008
694: stonewalling pairs accessed: 19632
445: stonewalling pairs accessed: 19387
496: stonewalling pairs accessed: 19539
432: stonewalling pairs accessed: 19347
19: stonewalling pairs accessed: 19372
31: stonewalling pairs accessed: 20043
22: stonewalling pairs accessed: 19745
1: stonewalling pairs accessed: 15273
689: stonewalling pairs accessed: 20261
434: stonewalling pairs accessed: 19317
505: stonewalling pairs accessed: 19556
501: stonewalling pairs accessed: 19810
502: stonewalling pairs accessed: 20156
438: stonewalling pairs accessed: 19089
692: stonewalling pairs accessed: 20255
439: stonewalling pairs accessed: 19582
693: stonewalling pairs accessed: 19426
5: stonewalling pairs accessed: 15123
700: stonewalling pairs accessed: 19869
21: stonewalling pairs accessed: 20071
444: stonewalling pairs accessed: 19938
511: stonewalling pairs accessed: 20164
691: stonewalling pairs accessed: 19994
620: stonewalling pairs accessed: 23199
508: stonewalling pairs accessed: 20163
690: stonewalling pairs accessed: 19655
591: stonewalling pairs accessed: 20852
291: stonewalling pairs accessed: 19951
442: stonewalling pairs accessed: 19927
796: stonewalling pairs accessed: 20787
353: stonewalling pairs accessed: 19569
623: stonewalling pairs accessed: 23505
437: stonewalling pairs accessed: 18870
440: stonewalling pairs accessed: 19163
431: stonewalling pairs accessed: 23550
17: stonewalling pairs accessed: 19409
16: stonewalling pairs accessed: 19182
4: stonewalling pairs accessed: 16041
584: stonewalling pairs accessed: 20812
298: stonewalling pairs accessed: 20450
216: stonewalling pairs accessed: 19805
535: stonewalling pairs accessed: 20085
418: stonewalling pairs accessed: 23153
504: stonewalling pairs accessed: 20161
829: stonewalling pairs accessed: 20127
784: stonewalling pairs accessed: 20784
357: stonewalling pairs accessed: 20126
703: stonewalling pairs accessed: 20354
611: stonewalling pairs accessed: 22933
528: stonewalling pairs accessed: 19713
436: stonewalling pairs accessed: 18675
509: stonewalling pairs accessed: 19780
365: stonewalling pairs accessed: 19736
688: stonewalling pairs accessed: 19814
621: stonewalling pairs accessed: 23499
680: stonewalling pairs accessed: 22583
587: stonewalling pairs accessed: 20877
27: stonewalling pairs accessed: 19410
695: stonewalling pairs accessed: 19757
58: stonewalling pairs accessed: 20329
0: stonewalling pairs accessed: 14999
28: stonewalling pairs accessed: 19856
752: stonewalling pairs accessed: 20482
12: stonewalling pairs accessed: 15069
292: stonewalling pairs accessed: 19998
112: stonewalling pairs accessed: 20229
209: stonewalling pairs accessed: 19376
537: stonewalling pairs accessed: 20008
429: stonewalling pairs accessed: 23265
510: stonewalling pairs accessed: 20011
820: stonewalling pairs accessed: 20270
795: stonewalling pairs accessed: 20309
358: stonewalling pairs accessed: 19695
678: stonewalling pairs accessed: 23105
615: stonewalling pairs accessed: 22942
289: stonewalling pairs accessed: 19987
223: stonewalling pairs accessed: 19301
563: stonewalling pairs accessed: 22520
427: stonewalling pairs accessed: 23081
506: stonewalling pairs accessed: 19744
817: stonewalling pairs accessed: 20044
793: stonewalling pairs accessed: 20486
362: stonewalling pairs accessed: 19308
685: stonewalling pairs accessed: 23102
613: stonewalling pairs accessed: 22695
296: stonewalling pairs accessed: 20042
217: stonewalling pairs accessed: 20290
541: stonewalling pairs accessed: 20256
426: stonewalling pairs accessed: 23435
507: stonewalling pairs accessed: 19936
824: stonewalling pairs accessed: 20264
766: stonewalling pairs accessed: 20589
364: stonewalling pairs accessed: 19590
673: stonewalling pairs accessed: 22939
580: stonewalling pairs accessed: 19841
301: stonewalling pairs accessed: 20718
536: stonewalling pairs accessed: 20892
428: stonewalling pairs accessed: 23553
822: stonewalling pairs accessed: 19593
791: stonewalling pairs accessed: 19995
366: stonewalling pairs accessed: 19828
687: stonewalling pairs accessed: 23113
582: stonewalling pairs accessed: 20853
307: stonewalling pairs accessed: 20406
567: stonewalling pairs accessed: 22473
423: stonewalling pairs accessed: 22911
830: stonewalling pairs accessed: 20263
797: stonewalling pairs accessed: 20372
352: stonewalling pairs accessed: 19890
654: stonewalling pairs accessed: 20210
617: stonewalling pairs accessed: 22975
317: stonewalling pairs accessed: 20576
572: stonewalling pairs accessed: 22992
789: stonewalling pairs accessed: 20490
29: stonewalling pairs accessed: 20075
297: stonewalling pairs accessed: 20290
123: stonewalling pairs accessed: 22606
534: stonewalling pairs accessed: 20891
417: stonewalling pairs accessed: 22813
831: stonewalling pairs accessed: 19500
790: stonewalling pairs accessed: 20772
363: stonewalling pairs accessed: 19484
679: stonewalling pairs accessed: 22282
48: stonewalling pairs accessed: 19245
156: stonewalling pairs accessed: 23559
312: stonewalling pairs accessed: 20083
208: stonewalling pairs accessed: 19956
574: stonewalling pairs accessed: 22833
441: stonewalling pairs accessed: 19596
476: stonewalling pairs accessed: 23540
753: stonewalling pairs accessed: 19993
367: stonewalling pairs accessed: 20107
576: stonewalling pairs accessed: 19865
63: stonewalling pairs accessed: 20350
303: stonewalling pairs accessed: 20687
212: stonewalling pairs accessed: 20275
512: stonewalling pairs accessed: 20124
430: stonewalling pairs accessed: 23395
792: stonewalling pairs accessed: 20229
356: stonewalling pairs accessed: 19566
55: stonewalling pairs accessed: 19716
272: stonewalling pairs accessed: 19715
531: stonewalling pairs accessed: 19641
676: stonewalling pairs accessed: 22142
53: stonewalling pairs accessed: 20042
306: stonewalling pairs accessed: 19959
513: stonewalling pairs accessed: 22167
49: stonewalling pairs accessed: 19354
300: stonewalling pairs accessed: 20483
579: stonewalling pairs accessed: 20873
3: stonewalling pairs accessed: 15177
287: stonewalling pairs accessed: 19484
218: stonewalling pairs accessed: 19628
20: stonewalling pairs accessed: 19564
295: stonewalling pairs accessed: 20585
118: stonewalling pairs accessed: 22500
526: stonewalling pairs accessed: 22860
765: stonewalling pairs accessed: 20028
59: stonewalling pairs accessed: 20101
316: stonewalling pairs accessed: 20491
788: stonewalling pairs accessed: 20261
9: stonewalling pairs accessed: 16070
309: stonewalling pairs accessed: 20182
794: stonewalling pairs accessed: 20779
261: stonewalling pairs accessed: 23124
762: stonewalling pairs accessed: 20860
275: stonewalling pairs accessed: 20241
755: stonewalling pairs accessed: 20366
279: stonewalling pairs accessed: 19818
759: stonewalling pairs accessed: 20819
756: stonewalling pairs accessed: 20857
889: stonewalling pairs accessed: 20128
521: stonewalling pairs accessed: 22543
798: stonewalling pairs accessed: 20783
360: stonewalling pairs accessed: 19924
583: stonewalling pairs accessed: 20507
293: stonewalling pairs accessed: 20194
422: stonewalling pairs accessed: 23154
761: stonewalling pairs accessed: 20467
610: stonewalling pairs accessed: 22606
18: stonewalling pairs accessed: 19778
318: stonewalling pairs accessed: 20233
115: stonewalling pairs accessed: 22752
214: stonewalling pairs accessed: 19943
573: stonewalling pairs accessed: 22647
757: stonewalling pairs accessed: 20063
853: stonewalling pairs accessed: 22901
682: stonewalling pairs accessed: 23119
313: stonewalling pairs accessed: 20236
98: stonewalling pairs accessed: 20312
517: stonewalling pairs accessed: 22856
424: stonewalling pairs accessed: 23379
787: stonewalling pairs accessed: 19973
24: stonewalling pairs accessed: 19589
277: stonewalling pairs accessed: 19832
568: stonewalling pairs accessed: 22975
290: stonewalling pairs accessed: 19848
299: stonewalling pairs accessed: 20724
686: stonewalling pairs accessed: 22696
315: stonewalling pairs accessed: 20438
108: stonewalling pairs accessed: 20178
577: stonewalling pairs accessed: 19968
158: stonewalling pairs accessed: 23241
107: stonewalling pairs accessed: 19935
529: stonewalling pairs accessed: 19906
368: stonewalling pairs accessed: 19379
581: stonewalling pairs accessed: 20323
542: stonewalling pairs accessed: 20401
210: stonewalling pairs accessed: 19689
622: stonewalling pairs accessed: 23234
152: stonewalling pairs accessed: 23517
561: stonewalling pairs accessed: 22091
270: stonewalling pairs accessed: 23201
624: stonewalling pairs accessed: 19403
213: stonewalling pairs accessed: 19624
569: stonewalling pairs accessed: 22388
826: stonewalling pairs accessed: 19823
631: stonewalling pairs accessed: 19634
522: stonewalling pairs accessed: 22618
636: stonewalling pairs accessed: 20352
532: stonewalling pairs accessed: 19917
425: stonewalling pairs accessed: 23055
674: stonewalling pairs accessed: 22678
104: stonewalling pairs accessed: 20296
543: stonewalling pairs accessed: 20881
382: stonewalling pairs accessed: 20447
100: stonewalling pairs accessed: 19758
285: stonewalling pairs accessed: 20087
646: stonewalling pairs accessed: 19805
159: stonewalling pairs accessed: 23657
310: stonewalling pairs accessed: 20570
93: stonewalling pairs accessed: 23607
886: stonewalling pairs accessed: 20736
147: stonewalling pairs accessed: 23508
374: stonewalling pairs accessed: 20465
190: stonewalling pairs accessed: 22709
308: stonewalling pairs accessed: 19956
40: stonewalling pairs accessed: 20535
575: stonewalling pairs accessed: 22974
651: stonewalling pairs accessed: 19956
539: stonewalling pairs accessed: 20792
785: stonewalling pairs accessed: 20457
648: stonewalling pairs accessed: 20126
99: stonewalling pairs accessed: 19392
248: stonewalling pairs accessed: 20065
903: stonewalling pairs accessed: 19754
856: stonewalling pairs accessed: 23207
499: stonewalling pairs accessed: 19836
885: stonewalling pairs accessed: 20468
381: stonewalling pairs accessed: 20435
383: stonewalling pairs accessed: 20451
6: stonewalling pairs accessed: 15601
267: stonewalling pairs accessed: 23586
880: stonewalling pairs accessed: 19914
278: stonewalling pairs accessed: 20264
215: stonewalling pairs accessed: 20286
684: stonewalling pairs accessed: 22821
627: stonewalling pairs accessed: 19676
221: stonewalling pairs accessed: 19776
873: stonewalling pairs accessed: 22535
264: stonewalling pairs accessed: 23293
80: stonewalling pairs accessed: 22215
250: stonewalling pairs accessed: 19981
412: stonewalling pairs accessed: 23357
288: stonewalling pairs accessed: 19732
819: stonewalling pairs accessed: 19600
652: stonewalling pairs accessed: 20209
626: stonewalling pairs accessed: 19886
47: stonewalling pairs accessed: 20496
274: stonewalling pairs accessed: 19746
500: stonewalling pairs accessed: 19938
282: stonewalling pairs accessed: 20073
862: stonewalling pairs accessed: 23383
656: stonewalling pairs accessed: 23607
625: stonewalling pairs accessed: 19677
176: stonewalling pairs accessed: 22587
612: stonewalling pairs accessed: 22469
372: stonewalling pairs accessed: 20457
87: stonewalling pairs accessed: 23601
113: stonewalling pairs accessed: 22971
514: stonewalling pairs accessed: 22216
262: stonewalling pairs accessed: 23456
754: stonewalling pairs accessed: 20444
180: stonewalling pairs accessed: 22488
249: stonewalling pairs accessed: 20375
562: stonewalling pairs accessed: 22635
717: stonewalling pairs accessed: 20674
189: stonewalling pairs accessed: 23125
523: stonewalling pairs accessed: 22683
864: stonewalling pairs accessed: 22401
355: stonewalling pairs accessed: 19404
518: stonewalling pairs accessed: 22693
478: stonewalling pairs accessed: 23308
530: stonewalling pairs accessed: 20077
52: stonewalling pairs accessed: 20333
520: stonewalling pairs accessed: 22356
900: stonewalling pairs accessed: 19641
888: stonewalling pairs accessed: 20772
370: stonewalling pairs accessed: 19581
642: stonewalling pairs accessed: 18930
260: stonewalling pairs accessed: 23263
110: stonewalling pairs accessed: 20000
850: stonewalling pairs accessed: 22405
377: stonewalling pairs accessed: 19719
380: stonewalling pairs accessed: 20258
578: stonewalling pairs accessed: 20293
406: stonewalling pairs accessed: 23258
660: stonewalling pairs accessed: 23510
105: stonewalling pairs accessed: 19427
126: stonewalling pairs accessed: 22762
909: stonewalling pairs accessed: 19895
56: stonewalling pairs accessed: 19887
319: stonewalling pairs accessed: 20585
89: stonewalling pairs accessed: 22945
409: stonewalling pairs accessed: 23501
33: stonewalling pairs accessed: 19943
179: stonewalling pairs accessed: 22435
265: stonewalling pairs accessed: 23588
85: stonewalling pairs accessed: 22723
238: stonewalling pairs accessed: 19905
479: stonewalling pairs accessed: 23546
898: stonewalling pairs accessed: 20172
718: stonewalling pairs accessed: 19963
881: stonewalling pairs accessed: 20205
675: stonewalling pairs accessed: 22935
638: stonewalling pairs accessed: 20384
191: stonewalling pairs accessed: 23135
266: stonewalling pairs accessed: 23153
92: stonewalling pairs accessed: 23142
243: stonewalling pairs accessed: 19710
470: stonewalling pairs accessed: 23104
904: stonewalling pairs accessed: 20160
893: stonewalling pairs accessed: 20451
635: stonewalling pairs accessed: 20306
155: stonewalling pairs accessed: 23448
224: stonewalling pairs accessed: 19515
910: stonewalling pairs accessed: 20177
857: stonewalling pairs accessed: 23391
375: stonewalling pairs accessed: 20108
36: stonewalling pairs accessed: 19937
144: stonewalling pairs accessed: 23373
228: stonewalling pairs accessed: 20127
566: stonewalling pairs accessed: 22999
464: stonewalling pairs accessed: 22323
902: stonewalling pairs accessed: 20160
711: stonewalling pairs accessed: 20331
30: stonewalling pairs accessed: 19844
150: stonewalling pairs accessed: 23509
276: stonewalling pairs accessed: 19522
83: stonewalling pairs accessed: 22752
233: stonewalling pairs accessed: 19627
519: stonewalling pairs accessed: 22857
407: stonewalling pairs accessed: 23601
474: stonewalling pairs accessed: 23535
911: stonewalling pairs accessed: 19556
832: stonewalling pairs accessed: 22645
705: stonewalling pairs accessed: 20027
770: stonewalling pairs accessed: 22910
887: stonewalling pairs accessed: 20250
373: stonewalling pairs accessed: 19936
650: stonewalling pairs accessed: 19610
634: stonewalling pairs accessed: 20237
43: stonewalling pairs accessed: 19978
157: stonewalling pairs accessed: 23654
302: stonewalling pairs accessed: 20274
109: stonewalling pairs accessed: 20319
241: stonewalling pairs accessed: 19720
533: stonewalling pairs accessed: 20077
410: stonewalling pairs accessed: 23601
488: stonewalling pairs accessed: 20136
906: stonewalling pairs accessed: 20180
821: stonewalling pairs accessed: 19378
708: stonewalling pairs accessed: 20066
772: stonewalling pairs accessed: 22308
869: stonewalling pairs accessed: 22789
359: stonewalling pairs accessed: 20076
671: stonewalling pairs accessed: 24194
614: stonewalling pairs accessed: 22820
50: stonewalling pairs accessed: 19498
146: stonewalling pairs accessed: 23044
284: stonewalling pairs accessed: 19877
95: stonewalling pairs accessed: 23595
237: stonewalling pairs accessed: 20207
525: stonewalling pairs accessed: 22586
435: stonewalling pairs accessed: 19867
467: stonewalling pairs accessed: 22768
896: stonewalling pairs accessed: 19513
844: stonewalling pairs accessed: 23324
716: stonewalling pairs accessed: 20301
786: stonewalling pairs accessed: 20783
861: stonewalling pairs accessed: 22904
361: stonewalling pairs accessed: 20103
647: stonewalling pairs accessed: 20206
590: stonewalling pairs accessed: 20392
32: stonewalling pairs accessed: 19536
183: stonewalling pairs accessed: 23121
283: stonewalling pairs accessed: 20231
125: stonewalling pairs accessed: 23352
253: stonewalling pairs accessed: 20196
524: stonewalling pairs accessed: 23091
413: stonewalling pairs accessed: 23623
487: stonewalling pairs accessed: 19801
899: stonewalling pairs accessed: 19353
823: stonewalling pairs accessed: 19807
710: stonewalling pairs accessed: 20266
767: stonewalling pairs accessed: 20812
871: stonewalling pairs accessed: 22364
378: stonewalling pairs accessed: 19852
666: stonewalling pairs accessed: 24205
148: stonewalling pairs accessed: 23040
257: stonewalling pairs accessed: 23070
114: stonewalling pairs accessed: 22485
538: stonewalling pairs accessed: 20588
416: stonewalling pairs accessed: 22511
704: stonewalling pairs accessed: 19820
783: stonewalling pairs accessed: 20716
891: stonewalling pairs accessed: 20771
649: stonewalling pairs accessed: 20208
185: stonewalling pairs accessed: 23119
94: stonewalling pairs accessed: 23154
714: stonewalling pairs accessed: 20677
764: stonewalling pairs accessed: 20668
863: stonewalling pairs accessed: 23572
184: stonewalling pairs accessed: 22703
96: stonewalling pairs accessed: 19544
712: stonewalling pairs accessed: 19852
890: stonewalling pairs accessed: 20478
26: stonewalling pairs accessed: 20060
178: stonewalling pairs accessed: 22270
281: stonewalling pairs accessed: 19655
88: stonewalling pairs accessed: 22571
201: stonewalling pairs accessed: 23285
404: stonewalling pairs accessed: 22878
469: stonewalling pairs accessed: 23217
713: stonewalling pairs accessed: 20175
892: stonewalling pairs accessed: 20316
665: stonewalling pairs accessed: 23598
37: stonewalling pairs accessed: 19822
145: stonewalling pairs accessed: 22773
256: stonewalling pairs accessed: 22674
102: stonewalling pairs accessed: 19934
251: stonewalling pairs accessed: 20668
411: stonewalling pairs accessed: 23099
482: stonewalling pairs accessed: 20203
825: stonewalling pairs accessed: 19520
894: stonewalling pairs accessed: 20628
698: stonewalling pairs accessed: 19973
46: stonewalling pairs accessed: 20144
160: stonewalling pairs accessed: 19171
258: stonewalling pairs accessed: 22746
106: stonewalling pairs accessed: 19756
252: stonewalling pairs accessed: 19865
414: stonewalling pairs accessed: 23240
483: stonewalling pairs accessed: 19486
837: stonewalling pairs accessed: 22777
715: stonewalling pairs accessed: 19843
866: stonewalling pairs accessed: 22398
702: stonewalling pairs accessed: 19943
44: stonewalling pairs accessed: 20537
162: stonewalling pairs accessed: 19869
90: stonewalling pairs accessed: 23173
236: stonewalling pairs accessed: 19928
481: stonewalling pairs accessed: 19851
852: stonewalling pairs accessed: 22624
664: stonewalling pairs accessed: 23420
42: stonewalling pairs accessed: 19769
169: stonewalling pairs accessed: 19708
103: stonewalling pairs accessed: 20233
239: stonewalling pairs accessed: 20219
851: stonewalling pairs accessed: 22499
668: stonewalling pairs accessed: 23546
188: stonewalling pairs accessed: 22811
91: stonewalling pairs accessed: 23595
246: stonewalling pairs accessed: 19631
883: stonewalling pairs accessed: 20183
657: stonewalling pairs accessed: 23039
166: stonewalling pairs accessed: 19537
255: stonewalling pairs accessed: 19962
865: stonewalling pairs accessed: 22223
699: stonewalling pairs accessed: 20400
192: stonewalling pairs accessed: 22281
870: stonewalling pairs accessed: 22158
655: stonewalling pairs accessed: 20208
193: stonewalling pairs accessed: 22447
877: stonewalling pairs accessed: 22252
669: stonewalling pairs accessed: 23793
204: stonewalling pairs accessed: 22894
879: stonewalling pairs accessed: 23075
683: stonewalling pairs accessed: 22522
245: stonewalling pairs accessed: 20144
855: stonewalling pairs accessed: 22925
661: stonewalling pairs accessed: 23608
222: stonewalling pairs accessed: 20269
895: stonewalling pairs accessed: 20764
57: stonewalling pairs accessed: 19974
196: stonewalling pairs accessed: 23091
874: stonewalling pairs accessed: 23035
230: stonewalling pairs accessed: 20044
816: stonewalling pairs accessed: 19637
116: stonewalling pairs accessed: 22966
200: stonewalling pairs accessed: 23360
485: stonewalling pairs accessed: 19844
827: stonewalling pairs accessed: 20253
867: stonewalling pairs accessed: 22789
670: stonewalling pairs accessed: 24017
294: stonewalling pairs accessed: 20348
839: stonewalling pairs accessed: 23054
724: stonewalling pairs accessed: 24395
371: stonewalling pairs accessed: 20057
782: stonewalling pairs accessed: 22985
25: stonewalling pairs accessed: 19908
928: stonewalling pairs accessed: 23007
491: stonewalling pairs accessed: 20333
737: stonewalling pairs accessed: 23523
940: stonewalling pairs accessed: 24258
458: stonewalling pairs accessed: 22692
943: stonewalling pairs accessed: 24211
738: stonewalling pairs accessed: 23764
740: stonewalling pairs accessed: 23324
779: stonewalling pairs accessed: 22977
907: stonewalling pairs accessed: 20043
847: stonewalling pairs accessed: 23528
875: stonewalling pairs accessed: 22455
662: stonewalling pairs accessed: 23543
205: stonewalling pairs accessed: 23196
473: stonewalling pairs accessed: 23220
778: stonewalling pairs accessed: 22694
203: stonewalling pairs accessed: 23570
846: stonewalling pairs accessed: 23051
720: stonewalling pairs accessed: 23360
641: stonewalling pairs accessed: 20193
743: stonewalling pairs accessed: 23825
165: stonewalling pairs accessed: 19169
749: stonewalling pairs accessed: 23699
174: stonewalling pairs accessed: 20045
472: stonewalling pairs accessed: 23093
771: stonewalling pairs accessed: 22176
643: stonewalling pairs accessed: 19124
175: stonewalling pairs accessed: 19475
919: stonewalling pairs accessed: 20136
734: stonewalling pairs accessed: 23962
170: stonewalling pairs accessed: 20004
64: stonewalling pairs accessed: 20249
199: stonewalling pairs accessed: 23085
776: stonewalling pairs accessed: 22363
845: stonewalling pairs accessed: 23475
941: stonewalling pairs accessed: 23841
273: stonewalling pairs accessed: 20068
465: stonewalling pairs accessed: 22476
930: stonewalling pairs accessed: 22963
119: stonewalling pairs accessed: 22796
746: stonewalling pairs accessed: 24327
420: stonewalling pairs accessed: 22689
448: stonewalling pairs accessed: 22998
931: stonewalling pairs accessed: 23255
77: stonewalling pairs accessed: 23000
742: stonewalling pairs accessed: 24261
79: stonewalling pairs accessed: 23698
206: stonewalling pairs accessed: 23425
923: stonewalling pairs accessed: 19525
195: stonewalling pairs accessed: 22955
419: stonewalling pairs accessed: 22468
463: stonewalling pairs accessed: 23154
934: stonewalling pairs accessed: 23392
722: stonewalling pairs accessed: 23499
667: stonewalling pairs accessed: 23408
72: stonewalling pairs accessed: 22959
741: stonewalling pairs accessed: 23773
62: stonewalling pairs accessed: 20195
149: stonewalling pairs accessed: 23284
925: stonewalling pairs accessed: 19887
835: stonewalling pairs accessed: 23002
733: stonewalling pairs accessed: 24395
653: stonewalling pairs accessed: 20145
736: stonewalling pairs accessed: 23405
471: stonewalling pairs accessed: 23519
726: stonewalling pairs accessed: 24393
186: stonewalling pairs accessed: 22471
415: stonewalling pairs accessed: 23595
475: stonewalling pairs accessed: 23353
922: stonewalling pairs accessed: 20143
707: stonewalling pairs accessed: 19867
868: stonewalling pairs accessed: 22591
177: stonewalling pairs accessed: 22894
197: stonewalling pairs accessed: 22769
455: stonewalling pairs accessed: 23054
836: stonewalling pairs accessed: 23125
540: stonewalling pairs accessed: 19968
477: stonewalling pairs accessed: 23076
828: stonewalling pairs accessed: 19828
492: stonewalling pairs accessed: 19819
451: stonewalling pairs accessed: 22390
773: stonewalling pairs accessed: 22539
781: stonewalling pairs accessed: 22729
838: stonewalling pairs accessed: 22895
681: stonewalling pairs accessed: 22830
453: stonewalling pairs accessed: 23054
484: stonewalling pairs accessed: 19663
745: stonewalling pairs accessed: 23824
515: stonewalling pairs accessed: 22364
731: stonewalling pairs accessed: 24384
633: stonewalling pairs accessed: 20400
153: stonewalling pairs accessed: 23188
493: stonewalling pairs accessed: 19963
927: stonewalling pairs accessed: 19968
421: stonewalling pairs accessed: 23001
608: stonewalling pairs accessed: 22243
486: stonewalling pairs accessed: 20191
728: stonewalling pairs accessed: 23309
263: stonewalling pairs accessed: 22983
721: stonewalling pairs accessed: 24409
163: stonewalling pairs accessed: 19386
202: stonewalling pairs accessed: 23352
937: stonewalling pairs accessed: 23985
774: stonewalling pairs accessed: 22604
449: stonewalling pairs accessed: 22401
701: stonewalling pairs accessed: 20393
732: stonewalling pairs accessed: 23906
609: stonewalling pairs accessed: 22357
938: stonewalling pairs accessed: 24261
751: stonewalling pairs accessed: 24322
271: stonewalling pairs accessed: 23607
593: stonewalling pairs accessed: 19708
777: stonewalling pairs accessed: 22544
78: stonewalling pairs accessed: 23228
735: stonewalling pairs accessed: 23219
39: stonewalling pairs accessed: 19928
503: stonewalling pairs accessed: 19845
605: stonewalling pairs accessed: 19874
134: stonewalling pairs accessed: 23547
921: stonewalling pairs accessed: 19989
833: stonewalling pairs accessed: 23113
124: stonewalling pairs accessed: 22874
405: stonewalling pairs accessed: 23054
916: stonewalling pairs accessed: 20143
747: stonewalling pairs accessed: 23823
897: stonewalling pairs accessed: 19720
462: stonewalling pairs accessed: 22687
70: stonewalling pairs accessed: 23319
65: stonewalling pairs accessed: 22806
232: stonewalling pairs accessed: 19459
130: stonewalling pairs accessed: 23175
841: stonewalling pairs accessed: 22840
606: stonewalling pairs accessed: 20038
76: stonewalling pairs accessed: 23696
137: stonewalling pairs accessed: 23570
854: stonewalling pairs accessed: 23392
8: stonewalling pairs accessed: 15261
443: stonewalling pairs accessed: 19243
128: stonewalling pairs accessed: 22622
926: stonewalling pairs accessed: 20143
920: stonewalling pairs accessed: 19625
780: stonewalling pairs accessed: 22561
644: stonewalling pairs accessed: 19246
565: stonewalling pairs accessed: 22217
461: stonewalling pairs accessed: 23134
181: stonewalling pairs accessed: 23031
571: stonewalling pairs accessed: 22800
235: stonewalling pairs accessed: 20221
849: stonewalling pairs accessed: 22282
901: stonewalling pairs accessed: 19990
878: stonewalling pairs accessed: 22841
354: stonewalling pairs accessed: 19891
456: stonewalling pairs accessed: 22418
136: stonewalling pairs accessed: 24128
384: stonewalling pairs accessed: 23344
596: stonewalling pairs accessed: 20285
706: stonewalling pairs accessed: 20190
304: stonewalling pairs accessed: 19620
122: stonewalling pairs accessed: 23352
939: stonewalling pairs accessed: 23519
834: stonewalling pairs accessed: 22659
842: stonewalling pairs accessed: 23388
69: stonewalling pairs accessed: 23193
131: stonewalling pairs accessed: 23535
600: stonewalling pairs accessed: 20510
602: stonewalling pairs accessed: 20510
744: stonewalling pairs accessed: 24325
663: stonewalling pairs accessed: 24041
527: stonewalling pairs accessed: 23102
392: stonewalling pairs accessed: 23241
607: stonewalling pairs accessed: 20203
142: stonewalling pairs accessed: 24333
82: stonewalling pairs accessed: 22585
254: stonewalling pairs accessed: 20666
395: stonewalling pairs accessed: 23599
398: stonewalling pairs accessed: 23595
604: stonewalling pairs accessed: 20594
11: stonewalling pairs accessed: 16105
393: stonewalling pairs accessed: 23571
632: stonewalling pairs accessed: 19892
391: stonewalling pairs accessed: 22865
139: stonewalling pairs accessed: 24300
595: stonewalling pairs accessed: 20060
760: stonewalling pairs accessed: 20138
859: stonewalling pairs accessed: 23244
45: stonewalling pairs accessed: 19665
915: stonewalling pairs accessed: 20015
599: stonewalling pairs accessed: 20445
399: stonewalling pairs accessed: 22562
259: stonewalling pairs accessed: 22995
394: stonewalling pairs accessed: 23310
41: stonewalling pairs accessed: 19647
141: stonewalling pairs accessed: 23997
386: stonewalling pairs accessed: 22948
597: stonewalling pairs accessed: 20160
390: stonewalling pairs accessed: 23557
598: stonewalling pairs accessed: 20506
194: stonewalling pairs accessed: 22637
389: stonewalling pairs accessed: 23234
628: stonewalling pairs accessed: 20305
564: stonewalling pairs accessed: 22820
242: stonewalling pairs accessed: 20148
490: stonewalling pairs accessed: 19860
286: stonewalling pairs accessed: 20264
231: stonewalling pairs accessed: 20134
396: stonewalling pairs accessed: 23005
187: stonewalling pairs accessed: 22607
343: stonewalling pairs accessed: 20540
397: stonewalling pairs accessed: 23417
117: stonewalling pairs accessed: 22249
14: stonewalling pairs accessed: 16113
466: stonewalling pairs accessed: 22529
348: stonewalling pairs accessed: 19993
101: stonewalling pairs accessed: 19814
129: stonewalling pairs accessed: 23083
408: stonewalling pairs accessed: 23205
876: stonewalling pairs accessed: 23077
320: stonewalling pairs accessed: 19715
314: stonewalling pairs accessed: 20290
74: stonewalling pairs accessed: 23697
324: stonewalling pairs accessed: 20244
328: stonewalling pairs accessed: 20070
111: stonewalling pairs accessed: 20317
872: stonewalling pairs accessed: 23026
516: stonewalling pairs accessed: 22565
495: stonewalling pairs accessed: 19331
336: stonewalling pairs accessed: 19837
338: stonewalling pairs accessed: 20506
322: stonewalling pairs accessed: 20309
121: stonewalling pairs accessed: 23119
229: stonewalling pairs accessed: 19861
337: stonewalling pairs accessed: 19980
133: stonewalling pairs accessed: 23437
311: stonewalling pairs accessed: 19871
942: stonewalling pairs accessed: 24080
2: stonewalling pairs accessed: 15903
933: stonewalling pairs accessed: 23802
447: stonewalling pairs accessed: 19944
347: stonewalling pairs accessed: 20546
127: stonewalling pairs accessed: 23414
339: stonewalling pairs accessed: 19989
207: stonewalling pairs accessed: 23589
334: stonewalling pairs accessed: 20538
86: stonewalling pairs accessed: 23128
480: stonewalling pairs accessed: 20172
342: stonewalling pairs accessed: 20219
321: stonewalling pairs accessed: 19947
335: stonewalling pairs accessed: 19680
858: stonewalling pairs accessed: 22968
51: stonewalling pairs accessed: 19750
143: stonewalling pairs accessed: 23153
120: stonewalling pairs accessed: 23353
350: stonewalling pairs accessed: 20536
138: stonewalling pairs accessed: 23954
560: stonewalling pairs accessed: 21967
331: stonewalling pairs accessed: 20539
637: stonewalling pairs accessed: 19986
344: stonewalling pairs accessed: 20259
10: stonewalling pairs accessed: 15340
860: stonewalling pairs accessed: 23582
327: stonewalling pairs accessed: 20495
211: stonewalling pairs accessed: 19808
333: stonewalling pairs accessed: 20338
73: stonewalling pairs accessed: 23211
332: stonewalling pairs accessed: 20418
345: stonewalling pairs accessed: 20547
268: stonewalling pairs accessed: 23059
468: stonewalling pairs accessed: 22917
280: stonewalling pairs accessed: 20151
329: stonewalling pairs accessed: 20257
800: stonewalling pairs accessed: 22077
346: stonewalling pairs accessed: 20220
433: stonewalling pairs accessed: 19927
805: stonewalling pairs accessed: 22558
140: stonewalling pairs accessed: 23591
810: stonewalling pairs accessed: 22698
351: stonewalling pairs accessed: 19740
182: stonewalling pairs accessed: 22547
639: stonewalling pairs accessed: 19408
340: stonewalling pairs accessed: 20247
135: stonewalling pairs accessed: 23591
758: stonewalling pairs accessed: 20481
619: stonewalling pairs accessed: 22875
84: stonewalling pairs accessed: 23383
132: stonewalling pairs accessed: 24128
802: stonewalling pairs accessed: 22557
349: stonewalling pairs accessed: 20371
234: stonewalling pairs accessed: 19862
570: stonewalling pairs accessed: 22622
168: stonewalling pairs accessed: 19449
813: stonewalling pairs accessed: 22709
812: stonewalling pairs accessed: 22508
173: stonewalling pairs accessed: 19636
809: stonewalling pairs accessed: 22895
815: stonewalling pairs accessed: 23155
54: stonewalling pairs accessed: 20336
171: stonewalling pairs accessed: 19218
61: stonewalling pairs accessed: 19963
450: stonewalling pairs accessed: 23053
97: stonewalling pairs accessed: 20002
67: stonewalling pairs accessed: 22869
672: stonewalling pairs accessed: 22197
803: stonewalling pairs accessed: 22356
630: stonewalling pairs accessed: 20374
629: stonewalling pairs accessed: 19797
804: stonewalling pairs accessed: 22495
848: stonewalling pairs accessed: 22179
814: stonewalling pairs accessed: 22820
549: stonewalling pairs accessed: 22655
603: stonewalling pairs accessed: 20285
326: stonewalling pairs accessed: 20479
13: stonewalling pairs accessed: 15598
808: stonewalling pairs accessed: 22563
843: stonewalling pairs accessed: 23500
81: stonewalling pairs accessed: 22508
219: stonewalling pairs accessed: 20090
748: stonewalling pairs accessed: 24324
556: stonewalling pairs accessed: 22491
544: stonewalling pairs accessed: 20796
559: stonewalling pairs accessed: 23047
775: stonewalling pairs accessed: 22971
198: stonewalling pairs accessed: 23013
801: stonewalling pairs accessed: 22191
68: stonewalling pairs accessed: 23042
71: stonewalling pairs accessed: 22693
557: stonewalling pairs accessed: 22719
401: stonewalling pairs accessed: 22901
558: stonewalling pairs accessed: 22735
555: stonewalling pairs accessed: 22352
151: stonewalling pairs accessed: 23082
167: stonewalling pairs accessed: 19991
840: stonewalling pairs accessed: 23472
547: stonewalling pairs accessed: 22431
240: stonewalling pairs accessed: 19537
154: stonewalling pairs accessed: 23584
545: stonewalling pairs accessed: 22164
548: stonewalling pairs accessed: 22557
60: stonewalling pairs accessed: 20343
402: stonewalling pairs accessed: 23103
15: stonewalling pairs accessed: 14818
330: stonewalling pairs accessed: 20364
454: stonewalling pairs accessed: 22694
594: stonewalling pairs accessed: 19866
585: stonewalling pairs accessed: 20175
918: stonewalling pairs accessed: 19939
884: stonewalling pairs accessed: 20771
385: stonewalling pairs accessed: 22652
677: stonewalling pairs accessed: 22413
269: stonewalling pairs accessed: 23609
739: stonewalling pairs accessed: 24109
457: stonewalling pairs accessed: 22585
66: stonewalling pairs accessed: 23337
388: stonewalling pairs accessed: 23594
400: stonewalling pairs accessed: 22743
659: stonewalling pairs accessed: 23348
452: stonewalling pairs accessed: 22800
799: stonewalling pairs accessed: 20101
369: stonewalling pairs accessed: 20471
750: stonewalling pairs accessed: 23856
932: stonewalling pairs accessed: 23694
935: stonewalling pairs accessed: 23856
811: stonewalling pairs accessed: 23149
34: stonewalling pairs accessed: 19641
807: stonewalling pairs accessed: 22231
305: stonewalling pairs accessed: 19878
323: stonewalling pairs accessed: 20107
768: stonewalling pairs accessed: 20773
727: stonewalling pairs accessed: 23100
658: stonewalling pairs accessed: 23201
75: stonewalling pairs accessed: 23246
936: stonewalling pairs accessed: 24255
914: stonewalling pairs accessed: 19930
379: stonewalling pairs accessed: 20053
38: stonewalling pairs accessed: 20395
719: stonewalling pairs accessed: 20332
586: stonewalling pairs accessed: 20512
924: stonewalling pairs accessed: 19717
588: stonewalling pairs accessed: 20468
489: stonewalling pairs accessed: 19535
882: stonewalling pairs accessed: 20726
908: stonewalling pairs accessed: 20175
723: stonewalling pairs accessed: 24232
818: stonewalling pairs accessed: 19378
494: stonewalling pairs accessed: 20345
725: stonewalling pairs accessed: 23500
220: stonewalling pairs accessed: 20161
247: stonewalling pairs accessed: 19852
709: stonewalling pairs accessed: 20333
164: stonewalling pairs accessed: 19997
161: stonewalling pairs accessed: 19406
589: stonewalling pairs accessed: 20922
376: stonewalling pairs accessed: 20468
459: stonewalling pairs accessed: 23110
546: stonewalling pairs accessed: 22232
917: stonewalling pairs accessed: 19519
554: stonewalling pairs accessed: 23053
905: stonewalling pairs accessed: 19807
387: stonewalling pairs accessed: 23121
640: stonewalling pairs accessed: 19649
769: stonewalling pairs accessed: 22611
341: stonewalling pairs accessed: 20544
806: stonewalling pairs accessed: 22886
35: stonewalling pairs accessed: 19763
913: stonewalling pairs accessed: 19707
763: stonewalling pairs accessed: 20237
403: stonewalling pairs accessed: 23455
929: stonewalling pairs accessed: 23768
460: stonewalling pairs accessed: 22879
618: stonewalling pairs accessed: 23438
645: stonewalling pairs accessed: 19513
325: stonewalling pairs accessed: 20489
730: stonewalling pairs accessed: 23951
225: stonewalling pairs accessed: 19688
226: stonewalling pairs accessed: 20044
244: stonewalling pairs accessed: 19942
616: stonewalling pairs accessed: 23263
552: stonewalling pairs accessed: 23045
550: stonewalling pairs accessed: 23035
227: stonewalling pairs accessed: 19825
553: stonewalling pairs accessed: 22762
551: stonewalling pairs accessed: 22855
912: stonewalling pairs accessed: 20140
729: stonewalling pairs accessed: 23445
592: stonewalling pairs accessed: 19611
172: stonewalling pairs accessed: 20082
601: stonewalling pairs accessed: 20168
stonewalling pairs accessed min: 14818 max: 24409 -- min data: 0.6 GiB mean data: 0.9 GiB time: 300.1s
WARNING: Expected aggregate file size       = 1508768768000.
WARNING: Stat() of aggregate file size      = 1083162848768.
WARNING: Using actual aggregate bytes moved = 1083162848768.
WARNING: maybe caused by deadlineForStonewalling
write     2874.86    67314      340.79      45.91      45.91      16.94      342.31     0.126268   359.32     0   
Max Write: 2874.86 MiB/sec (3014.51 MB/sec)

Summary of all tests:
Operation   Max(MiB)   Min(MiB)  Mean(MiB)     StdDev   Max(OPs)   Min(OPs)  Mean(OPs)     StdDev    Mean(s) Stonewall(s) Stonewall(MiB) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt   blksiz    xsize aggs(MiB)   API RefNum
write        2874.86    2874.86    2874.86       0.00   64127.64   64127.64   64127.64       0.00  359.31613     300.11       3009.19     0    944  16    1   0     1        1         0    0  34000    47008    47008 1032984.6 MPIIO      0
Finished            : Sat Jul 11 07:51:55 2020
mdtest_easy_delete
-- started at 07/11/2020 08:42:36 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-r' '-F' '-P' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_easy' '-n' '1000000' '-u' '-L' '-a' 'POSIX' '--posix.odirect' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_easy-stonewall' '-N' '1'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 23.4%   Inodes: 905.7 Mi   Used Inodes: 19.0%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :     131043.094     131032.852     131038.452          2.263
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          1.004          1.004          1.004          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        901.733        901.663        901.695          0.016
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.996          0.996          0.996          0.000
-- finished at 07/11/2020 08:57:44 --

mdtest_easy_stat
-- started at 07/11/2020 08:31:12 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-T' '-F' '-P' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_easy' '-n' '1000000' '-u' '-L' '-a' 'POSIX' '--posix.odirect' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_easy-stonewall' '-N' '1'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 23.4%   Inodes: 905.7 Mi   Used Inodes: 19.0%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     774971.101     774663.357     774789.978         63.367
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        152.527        152.466        152.502          0.012
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 08:33:44 --

mdtest_easy_write
-- started at 07/11/2020 07:31:59 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-Y' '-C' '-F' '-P' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_easy' '-n' '1000000' '-u' '-L' '-a' 'POSIX' '--posix.odirect' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_easy-stonewall' '-N' '1' '-W' '300'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 22.8%   Inodes: 793.2 Mi   Used Inodes: 0.0%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files
Continue stonewall hit min: 62814 max: 125166 avg: 98901.4 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :     173959.117     173941.090     173949.643          4.114
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     311409.145             NA
   Tree creation             :          7.368          7.368          7.368          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        679.292        679.221        679.258          0.016
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        299.808             NA
   Tree creation             :          0.136          0.136          0.136          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 07:45:52 --

mdtest_hard_delete
-- started at 07/11/2020 09:01:22 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-r' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 23.4%   Inodes: 793.0 Mi   Used Inodes: 7.4%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :      79209.745      79202.131      79205.247          1.473
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.216          0.216          0.216          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :        781.056        780.980        781.025          0.015
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          4.640          4.640          4.640          0.000
-- finished at 07/11/2020 09:14:28 --

mdtest_hard_read
-- started at 07/11/2020 08:57:48 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-X' '-E' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 23.4%   Inodes: 793.0 Mi   Used Inodes: 7.4%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :     295104.057     295034.411     295059.373         13.942
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :        209.675        209.625        209.657          0.010
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 09:01:18 --

mdtest_hard_stat
-- started at 07/11/2020 08:39:54 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-T' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 23.4%   Inodes: 905.7 Mi   Used Inodes: 19.0%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files

SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :     390281.502     390135.310     390233.181         31.845
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :          0.000          0.000          0.000          0.000
   File stat                 :        158.564        158.504        158.524          0.013
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   Tree creation             :          0.000          0.000          0.000          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 08:42:33 --

mdtest_hard_write
-- started at 07/11/2020 07:51:58 --

mdtest-3.3.0+dev was launched with 944 total task(s) on 59 node(s)
Command line used: /opt/basis/rsc/io500/io500-app/bin/mdtest '-Y' '-C' '-t' '-F' '-P' '-w' '3901' '-e' '3901' '-d' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard' '-n' '1000000' '-x' '/lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr/mdt_hard-stonewall' '-a' 'POSIX' '-N' '1' '-W' '300'
Path: /lustre/iospb/io500-2020spring/datafiles/2020.07.11-05.26.49-scr
FS: 196.3 TiB   Used FS: 23.3%   Inodes: 905.9 Mi   Used Inodes: 12.4%

Nodemap: 11111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
V-0: Rank   0 Line  2166 Shifting ranks by 16 for each phase.
944 tasks, 944000000 files
Continue stonewall hit min: 57255 max: 65531 avg: 62617.3 


SUMMARY rate: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :      87968.282      87960.037      87964.185          1.721
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA     196548.624             NA
   Tree creation             :         18.288         18.288         18.288          0.000
   Tree removal              :          0.000          0.000          0.000          0.000

SUMMARY time: (of 1 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation             :        703.288        703.222        703.255          0.014
   File stat                 :          0.000          0.000          0.000          0.000
   File read                 :          0.000          0.000          0.000          0.000
   File removal              :          0.000          0.000          0.000          0.000
   File create (stonewall)   :             NA             NA        300.743             NA
   Tree creation             :          0.055          0.055          0.055          0.000
   Tree removal              :          0.000          0.000          0.000          0.000
-- finished at 07/11/2020 08:07:25 --

result_summary
[RESULT] BW   phase 1            ior_easy_write              126.112 GiB/s : time 363.24 seconds
[RESULT] IOPS phase 1         mdtest_easy_write              173.959 kiops : time 679.29 seconds
[RESULT] BW   phase 2            ior_hard_write                2.807 GiB/s : time 342.31 seconds
[RESULT] IOPS phase 2         mdtest_hard_write               87.968 kiops : time 703.29 seconds
[RESULT] IOPS phase 3                      find              151.510 kiops : time 1188.10 seconds
[RESULT] BW   phase 3             ior_easy_read              197.176 GiB/s : time 232.34 seconds
[RESULT] IOPS phase 4          mdtest_easy_stat              774.971 kiops : time 152.53 seconds
[RESULT] BW   phase 4             ior_hard_read                2.788 GiB/s : time 344.61 seconds
[RESULT] IOPS phase 5          mdtest_hard_stat              390.282 kiops : time 158.56 seconds
[RESULT] IOPS phase 6        mdtest_easy_delete              131.043 kiops : time 901.73 seconds
[RESULT] IOPS phase 7          mdtest_hard_read              295.104 kiops : time 209.68 seconds
[RESULT] IOPS phase 8        mdtest_hard_delete               79.210 kiops : time 789.91 seconds
[SCORE] Bandwidth 21.0048 GiB/s : IOPS 195.662 kiops : TOTAL 64.108