Home » HP Manuals » Storage Devices » HP Cisco MDS 9120 » Manual Viewer

HP Cisco MDS 9120 Cisco MDS 9000 Family CLI Configuration Guide, Release 3.x ( - Page 635

About Reordering PortChannel Frames, About Enabling In-Order Delivery

Add to My Manuals
Save this manual to your list of manuals

Page 635 highlights

Chapter 25 Configuring Fibre Channel Routing Services and Protocols In-Order Delivery Send documentation comments to [email protected] About Reordering PortChannel Frames When a link change occurs in a PortChannel, the frames for the same exchange or the same flow can switch from one path to another faster path. Figure 25-6 Switch 1 Link Congestion Delivery Frame 2 Frame 1 Switch 2 Old path New path Frame 4 Frame 3 85475 In Figure 25-6, the port of the old path (red dot) is congested. In this scenario, Frame 3 and Frame 4 can be delivered before Frame 1 and Frame 2. The the in-order delivery feature attempts to minimize the number of frames dropped during PortChannel link changes when the in-order delivery is enabled by sending a request to the remote switch on the PortChannel to flush all frames for this PortChannel. Note Both switches on the PortChannel must be running Cisco SAN-OS Release 3.0(1) for this IOD enhancement. For earlier releases, IOD waits for the switch latency period before sending new frames. When the in-order delivery guarantee feature is enabled and a PortChannel link change occurs, the frames crossing the PortChannel are treated as follows: • Frames using the old path are delivered before new frames are accepted. • The new frames are delivered through the new path after the switch latency drop period has elapsed and all old frames are flushed. Frames that cannot be delivered in order through the old path within the switch latency drop period are dropped. See the "Configuring the Drop Latency Time" section on page 25-17. About Enabling In-Order Delivery You can enable the in-order delivery feature for a specific VSAN or for the entire switch. By default, in-order delivery is disabled on switches in the Cisco MDS 9000 Family. Tip We recommend that you only enable this feature when devices that cannot handle any out-of-order frames are present in the switch. Load-balancing algorithms within the Cisco MDS 9000 Family ensure that frames are delivered in order during normal fabric operation. The load-balancing algorithms based on source FC ID, destination FC ID, and exchange ID are enforced in hardware without any performance degradation. However, if the fabric encounters a failure and this feature is enabled, the recovery will be delayed because of an intentional pausing of fabric forwarding to purge the fabric of resident frames that could potentially be forwarded out-of-order. OL-16184-01, Cisco MDS SAN-OS Release 3.x Cisco MDS 9000 Family CLI Configuration Guide 25-15

Section	Page
Cisco MDS 9000 Family CLI Configuration Guide, Release 3.x	1
Contents	3
New and Changed Information	53
Preface	61
Audience	61
Organization	61
Document Conventions	65
Related Documentation	67
Release Notes	67
Compatibility Information	67
Regulatory Compliance and Safety Information	67
Hardware Installation	67
Cisco Fabric Manager	68
Command-Line Interface	68
Part 1	75
Getting Started	75
Product Overview	77
Hardware Overview	77
Cisco MDS 9500 Series Multilayer Directors	78
Cisco MDS 9200 Series Fabric Switches	79
Cisco MDS 9216i Multiprotocol Fabric Switch	79
Cisco MDS 9222i, Cisco MDS 9216A and Cisco MDS 9216 Multilayer Fabric Switches	79
Cisco MDS 9100 Series Fixed Configuration Fabric Switches	80
Cisco SAN-OS Software Configuration	81
Tools for Software Configuration	81
CLI	81
Cisco MDS 9000 Fabric Manager	82
Software Configuration Overview	82
Basic Configuration	82
Advanced Configuration	83
Switch Configuration	83
Fabric Configuration	83
Security	84
IP Services	84
Intelligent Storage Services	84
Network and Switch Monitoring	84
Traffic Management	84
Before You Begin	85
About the Switch Prompt	86
Default Switch Roles	87
Using the CLI	87
CLI Command Modes	87
CLI Command Hierarchy	88
EXEC Mode Options	89
Configuration Mode	90
Configuration Mode Commands and Submodes	90
Displaying the Present Working Context	93
CLI Command Navigation	93
Command Completion	93
File System Completion	93
The no and Default Forms of Commands	94
CLI Command Configuration Options	94
Getting Help	94
Managing the Switch Configuration	95
Displaying the Switch Configuration	95
Saving a Configuration	98
Clearing a Configuration	98
Displaying Users	98
Sending Messages to Users	98
Using the ping and ping ipv6 Commands	99
Using the Extended ping and ping ipv6 Commands	99
Using traceroute and traceroute ipv6 Commands	100
Configuring Terminal Parameters	101
Setting the Terminal Session Timeout	101
Displaying Terminal Sessions	102
Clearing Terminal Sessions	102
Setting the Terminal Timeout	102
Setting the Terminal Type	103
Setting the Terminal Screen Length	103
Setting the Terminal Screen Width	103
Displaying Terminal Settings	103
Configuring the Switch Banner Message	103
Directing show Command Output to a File	104
Using CLI Variables	105
User-Defined CLI Session Variables	105
User-Defined CLI Persistent Variables	106
System-Defined Variables	107
Using Command Aliases	107
Defining Command Aliases	108
About Flash Devices	108
Internal bootflash	109
External CompactFlash (Slot0)	109
Formatting Flash Devices and File Systems	109
Initializing Internal bootflash	109
Formatting External CompactFlash	110
Using Switch File Systems	110
Specifying File Systems	111
Setting the Current Directory	111
Displaying the Current Directory	112
Displaying File Checksums	112
Listing the Files in a Directory	113
Creating a Directory	113
Deleting an Existing Directory	113
Moving Files	113
Copying Files	114
Deleting Files	115
Displaying File Contents	115
Saving Command Output to a File	116
Compressing and Uncompressing Files	116
Displaying the Last Lines in a File	116
Command Scripts	117
Executing Commands Specified in a Script	117
Using CLI Variables in Scripts	118
Setting the Delay Time	118
Part 2	121
Cisco MDS SAN-OS Installation and Switch Management	121
Obtaining and Installing Licenses	123
Licensing Terminology	123
Licensing Model	124
Licensing High Availability	130
Options to Install a License	130
Obtaining a Factory-Installed License	131
Performing a Manual Installation	131
Obtaining the License Key File	132
Installing the License Key File	132
Installing the License Key File to a Remote Location	134
Backing Up License Files	134
Identifying License Features in Use	134
Uninstalling Licenses	135
Updating Licenses	136
Grace Period Alerts	137
License Transfers Between Switches	138
Displaying License Information	138
On-Demand Port Activation Licensing	141
About On-Demand Port Activation Licensing	141
Port-Naming Conventions	142
Port Licensing	142
Default Configuration	144
License Status Definitions	148
Configuring Port Activation Licenses	150
Making a Port Eligible for a License	151
Acquiring a License for a Port	151
Moving Licenses Among Ports	152
On-Demand Port Activation License Example	153
Initial Configuration	157
Starting a Switch in the Cisco MDS 9000 Family	158
Initial Setup Routine	158
Preparing to Configure the Switch	159
Default Login	159
Setup Options	160
Assigning Setup Information	161
Configuring Out-of-Band Management	162
Configuring In-Band Management	166
Using the setup Command	169
Accessing the Switch	170
Assigning a Switch Name	171
Where Do You Go Next?	171
Verifying the Module Status	171
Configuring Date, Time, and Time Zone	172
Configuring the Time Zone	172
Adjusting for Daylight Saving Time or Summer Time	173
NTP Configuration	175
About NTP	175
NTP Configuration Guidelines	175
Configuring NTP	176
NTP CFS Distribution	179
Enabling NTP Distribution	179
Committing NTP Configuration Changes	179
Discarding NTP Configuration Changes	180
Releasing Fabric Session Lock	180
Database Merge Guidelines	180
NTP Session Status Verification	180
Management Interface Configuration	181
Obtaining Remote Management Access	181
Using the force Option During Shutdown	182
Default Gateway Configuration	182
Configuring the Default Gateway	183
Telnet Server Connection	183
Disabling a Telnet Connection	184
Configuring Console Port Settings	184
Verifying Console Port Settings	184
Configuring COM1 Port Settings	185
Verifying COM1 Port Settings	186
Configuring Modem Connections	186
Guidelines to Configure Modems	187
Enabling Modem Connections	188
Configuring the Initialization String	188
Configuring the Default Initialization String	189
Configuring a User-Specified Initialization String	190
Initializing a Modem in a Powered-On Switch	190
Verifying the Modem Connection Configuration	191
Configuring CDP	192
Clearing CDP Counters and Tables	193
Displaying CDP Information	193
Using the CFS Infrastructure	197
About CFS	197
Cisco SAN-OS Features Using CFS	198
CFS Features	198
CFS Protocol	199
CFS Distribution Scopes	199
CFS Distribution Modes	199
Uncoordinated Distribution	200
Coordinated Distribution	200
Unrestricted Uncoordinated Distributions	200
Disabling CFS Distribution on a Switch	200
Verifying CFS Distribution Status	201
CFS Application Requirements	201
Enabling CFS for an Application	201
Verifying Application Registration Status	201
Locking the Fabric	202
Verifying CFS Lock Status	202
Committing Changes	203
Discarding Changes	204
Saving the Configuration	204
Clearing a Locked Session	204
CFS Merge Support	204
Verifying CFS Merge Status	205
CFS Distribution over IP	207
Enabling CFS Over IP	208
Verifying the CFS Over IP Configuration	209
Configuring IP Multicast Address for CFS over IP	209
Verifying IP Multicast Address Configuration for CFS over IP	210
CFS Regions	211
About CFS Regions	211
Managing CFS Regions	212
Creating CFS Regions	212
Assigning Applications to CFS Regions	212
Moving an Application to a Different CFS Region	212
Removing an Application from a Region	213
Deleting CFS Regions	213
Default Settings	213
Software Images	215
About Software Images	215
Dependent Factors for Software Installation	216
Selecting the Correct Software Images for Cisco MDS 9100 Series Switches	216
Selecting the Correct Software Images for Cisco MDS 9200 Series Switches	216
Selecting the Correct Software Images for Cisco MDS 9500 Family Switches	216
Essential Upgrade Prerequisites	217
Software Upgrade Methods	219
Determining Software Compatibility	220
Automated Upgrades	220
Benefits of Using the install all Command	221
Recognizing Failure Cases	222
Using the install all Command	222
Upgrading Services Modules	225
Sample install all Commands	226
Upgrade Status Verification	234
Non-Disruptive Upgrades on Fabric and Modular Switches	234
Preparing for a Non-Disruptive Upgrade on Fabric and Modular Switches	235
Performing a Non-Disruptive Upgrade on a Fabric Switch	237
Viewing the Status of a Non-Disruptive Upgrade on a Fabric Switch	239
Troubleshooting a Non-Disruptive Upgrade on a Fabric Switch	239
Manual Upgrade on a Dual Supervisor Module Switch	240
Preparing for a Manual Installation	240
Upgrading a Loader	241
Upgrading the BIOS	243
Quick Upgrade	245
Downgrading from a Higher Release	245
Maintaining Supervisor Modules	246
Replacing Supervisor Modules	246
Migrating from Supervisor-1 Modules to Supervisor-2 Modules	246
Standby Supervisor Module Boot Variable Version	254
Standby Supervisor Module Bootflash Memory	254
Standby Supervisor Module Boot Alert	254
Installing Generation 2 Modules in Generation 1 Chassis	254
Replacing Modules	255
Default Settings	255
Working with Configuration Files	257
Managing Configuration Files	257
Displaying Configuration Files	257
Downloading Configuration Files to the Switch	258
From a Remote Server	258
From an External CompactFlash Disk (slot0:)	259
Saving Configuration Files to an External Device	259
To a Remote Server	259
To an External CompactFlash Disk (slot0:)	260
Saving the Running Configuration	260
Saving Startup Configurations in the Fabric	260
Unlocking the Startup Configuration File	261
Copying Configuration Files	261
Backing UpConfiguration Files	262
Rolling Back to a Previous Configuration	263
Restoring the Configured Redundancy Mode	263
Accessing File Systems on the Standby Supervisor Module	264
Deleting Configuration Files	264
Configuring High Availability	265
About High Availability	265
Switchover Mechanisms	266
HA Switchover Characteristics	266
Initiating a Switchover	266
Switchover Guidelines	267
Verifying Switchover Possibilities	267
Process Restartability	268
Synchronizing Supervisor Modules	268
Copying Boot Variable Images to the Standby Supervisor Module	268
Automatic Copying of Boot Variables	268
Verifying the Copied Boot Variables	268
Displaying HA Status Information	269
Managing System Hardware	273
Displaying Switch Hardware Inventory	273
Running Compact Flash Tests	276
Running the CompactFlash CRC Checksum Test On Demand	276
Enabling and Disabling the Automatic CompactFlash CRC Checksum Test	276
Setting the CompactFlash CRC Checksum Test Interval	277
Enabling and Disabling Failure Action at the Failure of a CompactFlash Checksum Test	277
Displaying the Frequency and Status of the CompactFlash CRC Checksum Test	277
Updating the CompactFlash Firmware	278
Updating the CompactFlash Firmware On Demand	278
Enabling and Disabling the CompactFlash Firmware Update	279
Setting the CompactFlash Firmware Update Interval	279
Enabling and Disabling Failure Action at the Failure of a CompactFlash Firmware Update	279
Displaying the Frequency and Status of CompactFlash Updates	280
Displaying CompactFlash CRC Test and Firmware Update Statistics	280
Displaying the Switch Serial Number	281
Displaying Power Usage Information	282
Power Supply Configuration Modes	282
Power Supply Configuration Guidelines	283
About Crossbar Management	285
Operational Considerations When Removing Crossbars	286
Graceful Shutdown of a Crossbar	286
Backward Compatibility for Generation 1 Modules in Cisco MDS 9513 Directors	287
About Module Temperature	287
Displaying Module Temperature	288
About Fan Modules	289
About Clock Modules	291
Displaying Environment Information	292
Default Settings	293
Managing Modules	295
About Modules	295
Supervisor Modules	296
Switching Modules	297
Services Modules	297
Verifying the Status of a Module	298
Checking the State of a Module	298
Connecting to a Module	299
Reloading Modules	300
Reloading a Switch	300
Power Cycling Modules	301
Reloading Switching Modules	301
Preserving Module Configuration	301
Purging Module Configuration	302
Powering Off Switching Modules	303
Identifying Module LEDs	303
EPLD Configuration	306
Upgrading EPLD Images	307
Displaying EPLD Versions	310
SSM Feature Support	312
Installing the SSI Boot Image on an SSM	312
Upgrading the SSI Boot Image on an SSM	313
SSI Boot Image Upgrade Considerations for the SSM	313
Verifying the SSI Boot Image	314
Configuring the SSI Image Boot Variable	317
Using the install ssi Command	319
Managing SSMs and Supervisor Modules	322
Considerations for Replacing SSMs and Supervisor Modules	322
Recovering an SSM After Replacing Corrupted CompactFlash Memory	322
Considerations for Upgrading and Downgrading Cisco MDS SAN-OS Releases	323
Default Settings	325
Part 3	327
Switch Configuration	327
Configuring Interfaces	329
Fibre Channel Interfaces	329
32-Port Switching Module Configuration Guidelines	330
About Interface Modes	331
E Port	332
F Port	332
FL Port	332
NP Ports	332
TL Port	333
TE Port	333
SD Port	333
ST Port	334
Fx Port	334
B Port	334
Auto Mode	334
N Port Identifier Virtualization	335
About Interface States	335
Administrative States	335
Operational States	335
Reason Codes	336
Configuring Fibre Channel Interfaces	339
Graceful Shutdown	340
Setting the Interface Administrative State	340
Configuring Interface Modes	341
Configuring System Default Port Mode F	341
Configuring Port Speeds	342
Autosensing	343
Enabling N Port Identifier Virtualization	343
About Interface Descriptions	343
Configuring the Interface Description	343
About Frame Encapsulation	344
About Receive Data Field Size	344
Configuring Receive Data Field Size	344
Identifying the Beacon LEDs	344
About Speed LEDs	345
About Beacon Mode	345
Configuring Beacon Mode	345
About Bit Error Thresholds	346
Switch Port Attribute Default Values	347
About SFP Transmitter Types	347
Displaying Interface Information	348
TL Ports for Private Loops	356
About TL Ports	357
About TL Port ALPA Caches	358
Displaying TL Port Information	358
Manually Inserting Entries into ALPA Cache	360
Displaying the ALPA Cache Contents	360
Clearing the ALPA Cache	360
Buffer Credits	360
About Buffer-to-Buffer Credits	360
Configuring Buffer-to-Buffer Credits	361
About Performance Buffers	362
Configuring Performance Buffers	362
About Extended BB_credits	362
Extended BB_credits on Generation 1 Switching Modules	363
Extended BB_credits on Generation 2 Switching Modules	364
Configuring Extended BB_credits	364
Displaying BB_Credit Information	365
Management Interfaces	365
About Management Interfaces	366
Configuring Management Interfaces	366
Displaying Management Interface Configuration	367
VSAN Interfaces	367
About VSAN Interfaces	367
Creating VSAN Interfaces	368
Displaying VSAN Interface Information	368
Default Settings	368
Configuring N Port Virtualization	369
About NPV	369
NPV Mode	371
NP Ports	372
NP Links	372
Internal FLOGI Parameters	372
Default Port Numbers	373
NPV Traffic Management	373
NPV Guidelines and Requirements	374
NPV Traffic Management Guidelines	375
Configuring NPV	375
SUMMARY STEPS	375
Configuring NPV Traffic Management	376
Configuring List of External Interfaces per Server Interfaces	376
Enable or Disable the Global Policy for Disruptive Load Balancing	377
Multiple VSAN Support	377
DPVM Configuration	377
NPV and Port Security	378
Verifying NPV	378
Verifying NPV Traffic Management	380
Configuring Generation 2 Switches and Modules	381
About Generation 2 Modules and Switches	381
Port Groups	382
Port Rate Modes	384
Dedicated Mode	386
Shared Mode	386
Dynamic Bandwidth Management	386
Out-of-Service Interfaces	387
Buffer Credit Allocation	387
Buffer Pools	388
BB_Credit Buffers for Switching Modules	389
48-port 4-Gbps Fibre Channel Module BB_Credit Buffers	389
24-port 4-Gbps Fibre Channel Module BB_Credit Buffers	391
18-Port Fibre Channel/4-Port GigabitEthernet Multiservice Module BB_Credit Buffers	392
12-Port 4-Gbps Switching Module BB_Credit Buffers	392
4-Port 10-Gbps Switching Module BB_Credit Buffers	393
BB_Credit Buffers for Fabric Switches	394
Cisco MDS 9134 Fabric Switch BB_Credit Buffers	394
Cisco MDS 9124 Fabric Switch BB_Credit Buffers	395
Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers	395
Extended BB_Credits	395
About Combining Generation 1 and Generation 2 Switching Modules	396
Port Indexes	396
PortChannels	398
Configuring Generation 2 Module Interface Shared Resources	400
Displaying Interface Capabilities	400
Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules	401
Migrating from Shared Mode to Dedicated Mode	401
Migrating from Dedicated Mode to Shared Mode	401
Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces	402
Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces	402
Configuring Port Speed	403
Configuring Rate Mode	404
Configuring Oversubscription Ratio Restrictions	406
Disabling Restrictions on Oversubscription Ratios	408
Oversubscription Ratio Restrictions Example	409
Enabling Restrictions on Oversubscription Ratios	411
Configuring Bandwidth Fairness	411
Enabling Bandwidth Fairness	412
Disabling Bandwidth Fairness	412
Upgrade or Downgrade Scenario	412
Taking Interfaces Out of Service	413
Releasing Shared Resources in a Port Group	414
Enabling the Buffer-to-Buffer State Change Number	414
Disabling ACL Adjacency Sharing for System Image Downgrade	415
Displaying SFP Diagnostic Information	415
Example Configurations	416
Configuring a 24-port 4-Gbps Fibre Channel Switching Module Example	416
Configuring a 48-port 4-Gbps Fibre Channel Switching Module Example	417
Default Settings	418
Configuring Trunking	419
About Trunking	419
Trunking Configuration Guidelines	420
Trunking Protocol	420
Enabling or Disabling the Trunking Protocol	421
About Trunk Mode	421
Configuring Trunk Mode	422
About Trunk-Allowed VSAN Lists	422
Configuring an Allowed-Active List of VSANs	424
Displaying Trunking Information	424
Default Settings	425
Configuring PortChannels	427
About PortChannels	427

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1,000
1,001
1,002
1,003
1,004
1,005
1,006
1,007
1,008
1,009
1,010
1,011
1,012
1,013
1,014
1,015
1,016
1,017
1,018
1,019
1,020
1,021
1,022
1,023
1,024
1,025
1,026
1,027
1,028
1,029
1,030
1,031
1,032
1,033
1,034
1,035
1,036
1,037
1,038
1,039
1,040
1,041
1,042
1,043
1,044
1,045
1,046
1,047
1,048
1,049
1,050
1,051
1,052
1,053
1,054
1,055
1,056
1,057
1,058
1,059
1,060
1,061
1,062
1,063
1,064
1,065
1,066
1,067
1,068
1,069
1,070
1,071
1,072
1,073
1,074
1,075
1,076
1,077
1,078
1,079
1,080
1,081
1,082
1,083
1,084
1,085
1,086
1,087
1,088
1,089
1,090
1,091
1,092
1,093
1,094
1,095
1,096
1,097
1,098
1,099
1,100
1,101
1,102
1,103
1,104
1,105
1,106
1,107
1,108
1,109
1,110
1,111
1,112
1,113
1,114
1,115
1,116
1,117
1,118
1,119
1,120
1,121
1,122
1,123
1,124
1,125
1,126
1,127
1,128
1,129
1,130
1,131
1,132
1,133
1,134
1,135
1,136
1,137
1,138
1,139
1,140
1,141
1,142
1,143
1,144
1,145
1,146
1,147
1,148
1,149
1,150
1,151
1,152
1,153
1,154
1,155
1,156
1,157
1,158
1,159
1,160
1,161
1,162
1,163
1,164
1,165
1,166
1,167
1,168
1,169
1,170
1,171
1,172
1,173
1,174
1,175
1,176
1,177
1,178
1,179
1,180
1,181
1,182
1,183
1,184
1,185
1,186
1,187
1,188
1,189
1,190
1,191
1,192
1,193
1,194
1,195
1,196
1,197
1,198
1,199
1,200
1,201
1,202
1,203
1,204
1,205
1,206
1,207
1,208
1,209
1,210
1,211
1,212
1,213
1,214
1,215
1,216
1,217
1,218
1,219
1,220
1,221
1,222
1,223
1,224
1,225
1,226
1,227
1,228
1,229
1,230
1,231
1,232
1,233
1,234
1,235
1,236
1,237
1,238
1,239
1,240
1,241
1,242
1,243
1,244
1,245
1,246
1,247
1,248
1,249
1,250
1,251
1,252
1,253
1,254
1,255
1,256
1,257
1,258
1,259
1,260
1,261
1,262
1,263
1,264
1,265
1,266
1,267
1,268
1,269
1,270
1,271
1,272
1,273
1,274
1,275
1,276
1,277
1,278
1,279
1,280
1,281
1,282
1,283
1,284
1,285
1,286
1,287
1,288
1,289
1,290
1,291
1,292
1,293
1,294
1,295
1,296
1,297
1,298
1,299
1,300
1,301
1,302
1,303
1,304
1,305
1,306
1,307
1,308
1,309
1,310
1,311
1,312
1,313
1,314
1,315
1,316
1,317
1,318
1,319
1,320
1,321
1,322
1,323
1,324
1,325
1,326
1,327
1,328
1,329
1,330
1,331
1,332
1,333
1,334
1,335
1,336
1,337
1,338
1,339
1,340
1,341
1,342
1,343
1,344
1,345
1,346
1,347
1,348
1,349
1,350
1,351
1,352
1,353
1,354
1,355
1,356
1,357
1,358
1,359
1,360
1,361
1,362
1,363
1,364
1,365
1,366
1,367
1,368
1,369
1,370
1,371
1,372
1,373
1,374
1,375
1,376
1,377
1,378
1,379
1,380
1,381
1,382
1,383
1,384
1,385
1,386
1,387
1,388
1,389
1,390
1,391
1,392
1,393
1,394
1,395
1,396
1,397
1,398
1,399
1,400
1,401
1,402
1,403
1,404
1,405
1,406
1,407
1,408
1,409
1,410
1,411
1,412
1,413
1,414
1,415
1,416
1,417
1,418
1,419
1,420
1,421
1,422
1,423
1,424
1,425
1,426
1,427
1,428
1,429
1,430
1,431
1,432
1,433
1,434
1,435
1,436
1,437
1,438
1,439
1,440
1,441
1,442
1,443
1,444

Match case Limit results 1 per page

Send documentation comments to [email protected]

25-15

Cisco MDS 9000 Family CLI Configuration Guide

OL-16184-01, Cisco MDS SAN-OS Release 3.x

Chapter 25

Configuring Fibre Channel Routing Services and Protocols

In-Order Delivery

About Reordering PortChannel Frames

When a link change occurs in a PortChannel, the frames for the same exchange or the same flow can

switch from one path to another faster path.

Figure 25-6

Link Congestion Delivery

Figure 25-6

, the port of the old path (red dot) is congested. In this scenario, Frame 3 and Frame 4 can

be delivered before Frame 1 and Frame 2.

The the in-order delivery feature attempts to minimize the number of frames dropped during

PortChannel link changes when the in-order delivery is enabled by sending a request to the remote switch

on the PortChannel to flush all frames for this PortChannel.

Note

Both switches on the PortChannel must be running Cisco SAN-OS Release 3.0(1) for this IOD

enhancement. For earlier releases, IOD waits for the switch latency period before sending new frames.

When the in-order delivery guarantee feature is enabled and a PortChannel link change occurs, the

frames crossing the PortChannel are treated as follows:

•

Frames using the old path are delivered before new frames are accepted.

•

The new frames are delivered through the new path after the switch latency drop period has elapsed

and all old frames are flushed.

Frames that cannot be delivered in order through the old path within the switch latency drop period are

dropped. See the

“Configuring the Drop Latency Time” section on page 25-17

About Enabling In-Order Delivery

You can enable the in-order delivery feature for a specific VSAN or for the entire switch. By default,

in-order delivery is disabled on switches in the Cisco MDS 9000 Family.

Tip

We recommend that you only enable this feature when devices that cannot handle any out-of-order

frames are present in the switch.

Load-balancing algorithms within the Cisco MDS 9000 Family ensure

that frames are delivered in order during normal fabric operation. The load-balancing algorithms based

on source FC ID, destination FC ID, and exchange ID are enforced in hardware without any performance

degradation. However, if the fabric encounters a failure and this feature is enabled, the recovery will be

delayed because of an intentional pausing of fabric forwarding to purge the fabric of resident frames that

could potentially be forwarded out-of-order.

Switch 1

Switch 2

Frame 2

Frame 1

Frame 4

Frame 3

Old path

New path

85475