Home » HP Manuals » Storage Devices » HP Cisco Nexus 5000 » Manual Viewer

HP Cisco Nexus 5000 Cisco MDS 9000 Family CLI Configuration Guide - Release 4. - Page 672

About Reordering PortChannel Frames, About Enabling In-Order Delivery

View all HP Cisco Nexus 5000 manuals

Add to My Manuals
Save this manual to your list of manuals

Page 672 highlights

In-Order Delivery Chapter 26 Configuring Fibre Channel Routing Services and Protocols Send documentation comments to [email protected] In Figure 26-5, the new path from Switch 1 to Switch 4 is faster. In this scenario, Frame 3 and Frame 4 may be delivered before Frame 1 and Frame 2. If the in-order guarantee feature is enabled, the frames within the network are treated as follows: • Frames in the network are delivered in the order in which they are transmitted. • Frames that cannot be delivered in order within the network latency drop period are dropped inside the network. About Reordering PortChannel Frames When a link change occurs in a PortChannel, the frames for the same exchange or the same flow can switch from one path to another faster path. Figure 26-6 Switch 1 Link Congestion Delivery Frame 2 Frame 1 Switch 2 Old path New path Frame 4 Frame 3 85475 In Figure 26-6, the port of the old path (red dot) is congested. In this scenario, Frame 3 and Frame 4 can be delivered before Frame 1 and Frame 2. The in-order delivery feature attempts to minimize the number of frames dropped during PortChannel link changes when the in-order delivery is enabled by sending a request to the remote switch on the PortChannel to flush all frames for this PortChannel. Note Both switches on the PortChannel must be running Cisco SAN-OS Release 3.0(1) for this IOD enhancement. For earlier releases, IOD waits for the switch latency period before sending new frames. When the in-order delivery guarantee feature is enabled and a PortChannel link change occurs, the frames crossing the PortChannel are treated as follows: • Frames using the old path are delivered before new frames are accepted. • The new frames are delivered through the new path after the switch latency drop period has elapsed and all old frames are flushed. Frames that cannot be delivered in order through the old path within the switch latency drop period are dropped. See the "Configuring the Drop Latency Time" section on page 26-17. About Enabling In-Order Delivery You can enable the in-order delivery feature for a specific VSAN or for the entire switch. By default, in-order delivery is disabled on switches in the Cisco MDS 9000 Family. 26-14 Cisco MDS 9000 Family CLI Configuration Guide OL-18084-01, Cisco MDS NX-OS Release 4.x

Section	Page
Cisco MDS 9000 Family CLI Configuration Guide, Release 4.x	1
Contents	3
New and Changed Information	57
Preface	61
Obtaining Documentation and Submitting a Service Request	68
Part 1	69
Getting Started	69
Product Overview	71
Hardware Overview	71
Cisco MDS 9500 Series Multilayer Directors	72
Cisco MDS 9200 Series Fabric Switches	73
Cisco MDS 9216i Multiprotocol Fabric Switch	73
Cisco MDS 9222i Multilayer Fabric Switch	73
Cisco MDS 9100 Series Fixed Configuration Fabric Switches	74
Cisco NX-OS Software Configuration	74
Tools for Software Configuration	75
CLI	75
Cisco MDS 9000 Fabric Manager	75
Software Configuration Overview	76
Basic Configuration	76
Advanced Configuration	76
Switch Configuration	77
Fabric Configuration	77
Security	77
IP Services	77
Intelligent Storage Services	78
Network and Switch Monitoring	78
Traffic Management	78
Before You Begin	79
About the Switch Prompt	80
Default Switch Roles	81
Using the CLI	81
CLI Command Modes	81
CLI Command Hierarchy	82
EXEC Mode Options	83
Configuration Mode	84
Configuration Mode Commands and Submodes	84
Displaying the Present Working Context	87
CLI Command Navigation	88
Command Completion	88
File System Completion	88
The no and Default Forms of Commands	88
CLI Command Configuration Options	89
Getting Help	89
Managing the Switch Configuration	90
Displaying the Switch Configuration	90
Saving a Configuration	93
Clearing a Configuration	93
Displaying Users	93
Sending Messages to Users	93
Using the ping and ping ipv6 Commands	93
Using the Extended ping and ping ipv6 Commands	94
Using traceroute and traceroute ipv6 Commands	95
Configuring Terminal Parameters	96
Setting the Terminal Session Timeout	96
Displaying Terminal Sessions	97
Clearing Terminal Sessions	97
Setting the Terminal Timeout	97
Setting the Terminal Type	97
Setting the Terminal Screen Length	98
Setting the Terminal Screen Width	98
Displaying Terminal Settings	98
Configuring the Switch Banner Message	98
Directing show Command Output to a File	99
Using CLI Variables	99
User-Defined CLI Persistent Variables	100
System-Defined Variables	101
Using Command Aliases	101
Defining Command Aliases	102
About Flash Devices	102
Internal bootflash	103
External CompactFlash (Slot0)	103
Formatting Flash Devices and File Systems	103
Initializing Internal bootflash	103
Formatting External CompactFlash	104
Using Switch File Systems	104
Specifying File Systems	105
Setting the Current Directory	106
Displaying the Current Directory	106
Displaying File Checksums	107
Listing the Files in a Directory	107
Creating a Directory	107
Deleting an Existing Directory	107
Moving Files	108
Copying Files	108
Deleting Files	109
Displaying File Contents	110
Saving Command Output to a File	110
Compressing and Uncompressing Files	110
Displaying the Last Lines in a File	111
Command Scripts	111
Executing Commands Specified in a Script	111
Using CLI Variables in Scripts	112
Setting the Delay Time	113
Part 2	115
Installation and Switch Management	115
Obtaining and Installing Licenses	117
Licensing Terminology	117
Licensing Model	118
Licensing High Availability	124
Options to Install a License	124
Obtaining a Factory-Installed License	125
Performing a Manual Installation	125
Obtaining the License Key File	126
Installing the License Key File	126
Installing the License Key File to a Remote Location	128
Backing Up License Files	128
Identifying License Features in Use	128
Uninstalling Licenses	129
Updating Licenses	130
Grace Period Alerts	131
License Transfers Between Switches	132
Displaying License Information	133
On-Demand Port Activation Licensing	135
About On-Demand Port Activation Licensing	135
Port-Naming Conventions	136
Port Licensing	136
Default Configuration	138
License Status Definitions	142
Configuring Port Activation Licenses	144
Making a Port Eligible for a License	145
Acquiring a License for a Port	145
Moving Licenses Among Ports	146
On-Demand Port Activation License Example	147
Initial Configuration	151
Starting a Switch in the Cisco MDS 9000 Family	152
Initial Setup Routine	153
Preparing to Configure the Switch	153
Default Login	154
Setup Options	154
Assigning Setup Information	155
Configuring Out-of-Band Management	156
Configuring In-Band Management	161
Using the setup Command	165
Accessing the Switch	165
Assigning a Switch Name	166
Where Do You Go Next?	166
Verifying the Module Status	167
Configuring Date, Time, and Time Zone	167
Configuring the Time Zone	168
Adjusting for Daylight Saving Time or Summer Time	168
NTP Configuration	169
About NTP	170
NTP Configuration Guidelines	170
Configuring NTP	171
NTP CFS Distribution	174
Enabling NTP Distribution	174
Committing NTP Configuration Changes	174
Discarding NTP Configuration Changes	175
Releasing Fabric Session Lock	175
Database Merge Guidelines	175
NTP Session Status Verification	175
Management Interface Configuration	176
Obtaining Remote Management Access	176
Using the force Option During Shutdown	177
Default Gateway Configuration	177
Configuring the Default Gateway	178
Telnet Server Connection	178
Disabling a Telnet Connection	179
Configuring Console Port Settings	179
Verifying Console Port Settings	179
Configuring COM1 Port Settings	180
Verifying COM1 Port Settings	181
Configuring Modem Connections	181
Guidelines to Configure Modems	182
Enabling Modem Connections	183
Configuring the Initialization String	183
Configuring the Default Initialization String	184
Configuring a User-Specified Initialization String	185
Initializing a Modem in a Powered-On Switch	185
Verifying the Modem Connection Configuration	186
Configuring CDP	187
Clearing CDP Counters and Tables	188
Displaying CDP Information	188
Configuring FlexAttach Virtual pWWN	193
About FlexAttach Virtual pWWN	193
FlexAttach Virtual pWWN Guidelines and Requirements	194
Configuring FlexAttach Virtual pWWN	194
Enabling FlexAttach Virtual pWWN	194
Automatically Enabling FlexAttach Virtual pWWN	194
Manually Enabling FlexAttach Virtual pWWN	195
Mapping pWWN to Virtual pWWN	195
Verifying FlexAttach Virtual pWWN	196
Verifying the End Device	196
Debugging FlexAttach Virtual pWWN	197
Security Settings for FlexAttach Virtual pWWN	197
FlexAttach Virtual pWWN CFS Distribution	198
Difference Between San Device Virtualization and FlexAttach Port Virtualization	198
Using the CFS Infrastructure	199
About CFS	199
Cisco MDS NX-OS Features Using CFS	200
CFS Features	200
CFS Protocol	201
CFS Distribution Scopes	201
CFS Distribution Modes	202
Uncoordinated Distribution	202
Coordinated Distribution	202
Unrestricted Uncoordinated Distributions	202
Disabling CFS Distribution on a Switch	202
Verifying CFS Distribution Status	203
CFS Application Requirements	203
Enabling CFS for an Application	204
Verifying Application Registration Status	204
Locking the Fabric	205
Verifying CFS Lock Status	205
Committing Changes	206
Discarding Changes	206
Saving the Configuration	206
Clearing a Locked Session	206
CFS Merge Support	207
Verifying CFS Merge Status	207
CFS Distribution over IP	209
Enabling CFS Over IP	211
Verifying the CFS Over IP Configuration	211
Configuring IP Multicast Address for CFS over IP	211
Verifying IP Multicast Address Configuration for CFS over IP	212
Configuring Static IP Peers for CFS over IP	212
Verifying Static IP Peer Configuration	214
CFS Regions	215
About CFS Regions	215
Managing CFS Regions	216
Creating CFS Regions	216
Assigning Applications to CFS Regions	216
Moving an Application to a Different CFS Region	216
Removing an Application from a Region	217
Deleting CFS Regions	217
Displaying CFS Regions	217
Default Settings	217
Software Images	219
About Software Images	219
Dependent Factors for Software Installation	220
Selecting the Correct Software Images for Cisco MDS 9100 Series Switches	220
Selecting the Correct Software Images for Cisco MDS 9200 Series Switches	220
Selecting the Correct Software Images for Cisco MDS 9500 Family Switches	220
Essential Upgrade Prerequisites	221
Software Upgrade Methods	223
Determining Software Compatibility	224
Automated Upgrades	225
Benefits of Using the install all Command	225
Recognizing Failure Cases	226
Using the install all Command	227
Upgrading Services Modules	230
Sample install all Commands	231
Upgrade Status Verification	238
Nondisruptive Upgrades on Fabric and Modular Switches	239
Preparing for a Nondisruptive Upgrade on Fabric and Modular Switches	239
Performing a Nondisruptive Upgrade on a Fabric Switch	242
Viewing the Status of a Nondisruptive Upgrade on a Fabric Switch	243
Troubleshooting a Nondisruptive Upgrade on a Fabric Switch	244
Manual Upgrade on a Dual Supervisor Module Switch	244
Preparing for a Manual Installation	245
Upgrading a Loader	246
Upgrading the BIOS	248
Quick Upgrade	249
Downgrading from a Higher Release	249
Maintaining Supervisor Modules	250
Replacing Supervisor Modules	251
Migrating from Supervisor-1 Modules to Supervisor-2 Modules	251
Standby Supervisor Module Boot Variable Version	251
Standby Supervisor Module Bootflash Memory	252
Standby Supervisor Module Boot Alert	252
Installing Generation 2 Modules in Generation 1 Chassis	252
Replacing Modules	252
Default Settings	253
Working with Configuration Files	255
Managing Configuration Files	255
Displaying Configuration Files	255
Downloading Configuration Files to the Switch	256
Using a File From a Remote Server	256
Using a File From an External CompactFlash Disk (slot0:)	257
Saving Configuration Files to an External Device	257
Saving to a Remote Server	257
Saving to an External CompactFlash Disk (slot0:)	258
Saving the Running Configuration	258
Saving Startup Configurations in the Fabric	258
Unlocking the Startup Configuration File	259
Copying Configuration Files	259
Backing Up Configuration Files	260
Rolling Back to a Previous Configuration	261
Restoring the Configured Redundancy Mode	261
Accessing File Systems on the Standby Supervisor Module	262
Deleting Configuration Files	262
Configuring High Availability	263
About High Availability	263
Switchover Mechanisms	264
HA Switchover Characteristics	264
Initiating a Switchover	264
Switchover Guidelines	265
Verifying Switchover Possibilities	265
Process Restartability	266
Synchronizing Supervisor Modules	266
Copying Boot Variable Images to the Standby Supervisor Module	266
Automatic Copying of Boot Variables	266
Verifying the Copied Boot Variables	266
Displaying HA Status Information	267
Displaying the System Uptime	269
Managing System Hardware	271
Displaying Switch Hardware Inventory	271
Running Compact Flash Tests	274
Running the CompactFlash CRC Checksum Test On Demand	274
Enabling and Disabling the Automatic CompactFlash CRC Checksum Test	274
Setting the CompactFlash CRC Checksum Test Interval	275
Enabling and Disabling Failure Action at the Failure of a CompactFlash Checksum Test	275
Displaying the Frequency and Status of the CompactFlash CRC Checksum Test	275
Updating the CompactFlash Firmware	276
Updating the CompactFlash Firmware On Demand	276
Enabling and Disabling the CompactFlash Firmware Update	277
Setting the CompactFlash Firmware Update Interval	277
Enabling and Disabling Failure Action at the Failure of a CompactFlash Firmware Update	277
Displaying the Frequency and Status of CompactFlash Updates	278
Displaying CompactFlash CRC Test and Firmware Update Statistics	278
Displaying the Switch Serial Number	279
Displaying Power Usage Information	280
Power Supply Configuration Modes	280
Power Supply Configuration Guidelines	281
About Crossbar Management	283
Operational Considerations When Removing Crossbars	284
Gracefully Shutting Down a Crossbar	285
Backward Compatibility for Generation 1 Modules in Cisco MDS 9513 Directors	285
About Module Temperature	286
Displaying Module Temperature	287
About Fan Modules	287
About Clock Modules	289
Displaying Environment Information	290
Default Settings	291
Managing Modules	293
About Modules	294
Supervisor Modules	294
Switching Modules	295
Services Modules	295
Verifying the Status of a Module	296
Checking the State of a Module	296
Connecting to a Module	297
Reloading Modules	298
Reloading a Switch	298
Power Cycling Modules	298
Reloading Switching Modules	299
Preserving the Module Configuration	299
Purging Module Configuration	300
Powering Off Switching Modules	301
Identifying Module LEDs	301
EPLD Configuration	304
Upgrading EPLD Images	305
Displaying EPLD Versions	308
Installing the SSI Boot Image	310
Upgrading the SSI Boot Image	311
SSI Boot Image Upgrade Considerations for the SSM	311
Verifying the SSI Boot Image	312
Configuring the SSI Image Boot Variable	315
Using the install ssi Command	317
Managing SSMs and Supervisor Modules	320
Configuring SSM Global Upgrade Delay	320
Considerations for Replacing SSMs and Supervisor Modules	320
Recovering an SSM After Replacing Corrupted CompactFlash Memory	321
Considerations for Upgrading and Downgrading Cisco MDS NX-OS Releases	321
Default Settings	323
Part 3	325
Switch Configuration	325
Configuring Interfaces	327
Fibre Channel Interfaces	327
Generation 1 Interfaces Configuration Guidelines	328
About Interface Modes	329
E Port	330
F Port	330
FL Port	330
NP Ports	330
TL Port	331
TE Port	331
TF Port	331
TNP Port	332
SD Port	332
ST Port	332
Fx Port	332
B Port	332
Auto Mode	333
N Port Identifier Virtualization	333
About Interface States	333
Administrative States	333
Operational States	334
Reason Codes	334
Configuring Fibre Channel Interfaces	337
Graceful Shutdown	338
Setting the Interface Administrative State	338
Configuring Interface Modes	339
Configuring System Default Port Mode F	339
Configuring Port Speeds	340
Autosensing	341
Specifying a Port Owner	341
Configuring Port Guard	342
Enabling N Port Identifier Virtualization	343
About Interface Descriptions	343
Configuring the Interface Description	343
About Frame Encapsulation	343
About Receive Data Field Size	344
Configuring Receive Data Field Size	344
Identifying the Beacon LEDs	344
About Speed LEDs	344
About Beacon Mode	345
Configuring Beacon Mode	345
About Bit Error Thresholds	345
Switch Port Attribute Default Values	346
About SFP Transmitter Types	347
Displaying Interface Information	348
TL Ports for Private Loops	356
About TL Ports	356
About TL Port ALPA Caches	358
Displaying TL Port Information	358
Manually Inserting Entries into ALPA Cache	360
Displaying the ALPA Cache Contents	360
Clearing the ALPA Cache	360
Buffer Credits	360
About Buffer-to-Buffer Credits	360
Configuring Buffer-to-Buffer Credits	361
About Performance Buffers	362
Configuring Performance Buffers	362
About Extended BB_credits	362
Extended BB_credits on Generation 1 Switching Modules	363
Extended BB_credits on Generation 2 and Generation 3 Switching Modules	364
Configuring Extended BB_credits	364
Displaying BB_Credit Information	365
Management Interfaces	366
About Management Interfaces	366
Configuring Management Interfaces	366
Displaying Management Interface Configuration	367
VSAN Interfaces	368
About VSAN Interfaces	368
Creating VSAN Interfaces	368
Displaying VSAN Interface Information	368
Default Settings	369
Configuring N Port Virtualization	371
About NPV	371
NPV Mode	373
NP Ports	374
NP Links	374
Internal FLOGI Parameters	374
Default Port Numbers	375
NPV CFS Distribution over IP	376
NPV Traffic Management	376
Auto	376
Traffic Map	376
Disruptive	377
Multiple VSAN Support	377
NPV Guidelines and Requirements	377
NPV Traffic Management Guidelines	378
Configuring NPV	378
SUMMARY STEPS	378
Configuring NPV Traffic Management	380
Configuring List of External Interfaces per Server Interface	380
Enable or Disable the Global Policy for Disruptive Load Balancing	380
DPVM Configuration	381
NPV and Port Security	381
Verifying NPV	381
Verifying NPV Traffic Management	383
Configuring Generation 2 and Generation 3 Switching Modules	385
About Generations of Modules and Switches	385
Port Groups and Port Rate Modes	387
Port Groups	387
Port Rate Modes	388
Dedicated Rate Mode	390
Shared Rate Mode	391
Dedicated Rate Mode Configurations for the 8-Gbps Modules	391
Dynamic Bandwidth Management	392
Out-of-Service Interfaces	392
Buffer Credit Allocation	392
Buffer Pools	393
BB_Credit Buffers for Switching Modules	395
48-Port 8-Gbps Fibre Channel Module BB_Credit Buffers	396
24-Port 8-Gbps Fibre Channel Module BB_Credit Buffers	397
4/44-Port 8-Gbps Host-Optimized Fibre Channel Module BB_Credit Buffers	398
48-Port 4-Gbps Fibre Channel Module BB_Credit Buffers	399
24-Port 4-Gbps Fibre Channel Module BB_Credit Buffers	401
18-Port Fibre Channel/4-Port Gigabit Ethernet Multiservice Module BB_Credit Buffers	402
12-Port 4-Gbps Switching Module BB_Credit Buffers	402
4-Port 10-Gbps Switching Module BB_Credit Buffers	404
BB_Credit Buffers for Fabric Switches	405
Cisco MDS 9134 Fabric Switch BB_Credit Buffers	405
Cisco MDS 9124 Fabric Switch BB_Credit Buffers	405
Cisco MDS 9222i Multiservice Modular Switch BB_Credit Buffers	406
Extended BB_Credits	406
Combining Generation 1, Generation 2, and Generation 3 Modules	406
Port Indexes	407
PortChannels	409
Configuring Module Interface Shared Resources	412
Displaying Interface Capabilities	413
Configuration Guidelines for 48-Port, 24-Port, and 4/44-Port 8-Gbps Fibre Channel Switching Modules	414
Migrating from Shared Mode to Dedicated Mode	414
Migrating from Dedicated Mode to Shared Mode	414
Configuration Guidelines for 48-Port and 24-Port 4-Gbps Fibre Channel Switching Modules	415
Migrating from Shared Mode to Dedicated Mode	415
Migrating from Dedicated Mode to Shared Mode	416
Configuration Guidelines for 12-Port 4-Gbps Switching Module Interfaces	416
Configuration Guidelines for 4-Port 10-Gbps Switching Module Interfaces	417
Configuring Port Speed	417
Configuring Rate Mode	419
Configuring Oversubscription Ratio Restrictions	424
Disabling Restrictions on Oversubscription Ratios	426
Oversubscription Ratio Restrictions Example	427
Enabling Restrictions on Oversubscription Ratios	429
Configuring Bandwidth Fairness	429
Enabling Bandwidth Fairness	430

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1,000
1,001
1,002
1,003
1,004
1,005
1,006
1,007
1,008
1,009
1,010
1,011
1,012
1,013
1,014
1,015
1,016
1,017
1,018
1,019
1,020
1,021
1,022
1,023
1,024
1,025
1,026
1,027
1,028
1,029
1,030
1,031
1,032
1,033
1,034
1,035
1,036
1,037
1,038
1,039
1,040
1,041
1,042
1,043
1,044
1,045
1,046
1,047
1,048
1,049
1,050
1,051
1,052
1,053
1,054
1,055
1,056
1,057
1,058
1,059
1,060
1,061
1,062
1,063
1,064
1,065
1,066
1,067
1,068
1,069
1,070
1,071
1,072
1,073
1,074
1,075
1,076
1,077
1,078
1,079
1,080
1,081
1,082
1,083
1,084
1,085
1,086
1,087
1,088
1,089
1,090
1,091
1,092
1,093
1,094
1,095
1,096
1,097
1,098
1,099
1,100
1,101
1,102
1,103
1,104
1,105
1,106
1,107
1,108
1,109
1,110
1,111
1,112
1,113
1,114
1,115
1,116
1,117
1,118
1,119
1,120
1,121
1,122
1,123
1,124
1,125
1,126
1,127
1,128
1,129
1,130
1,131
1,132
1,133
1,134
1,135
1,136
1,137
1,138
1,139
1,140
1,141
1,142
1,143
1,144
1,145
1,146
1,147
1,148
1,149
1,150
1,151
1,152
1,153
1,154
1,155
1,156
1,157
1,158
1,159
1,160
1,161
1,162
1,163
1,164
1,165
1,166
1,167
1,168
1,169
1,170
1,171
1,172
1,173
1,174
1,175
1,176
1,177
1,178
1,179
1,180
1,181
1,182
1,183
1,184
1,185
1,186
1,187
1,188
1,189
1,190
1,191
1,192
1,193
1,194
1,195
1,196
1,197
1,198
1,199
1,200
1,201
1,202
1,203
1,204
1,205
1,206
1,207
1,208
1,209
1,210
1,211
1,212
1,213
1,214
1,215
1,216
1,217
1,218
1,219
1,220
1,221
1,222
1,223
1,224
1,225
1,226
1,227
1,228
1,229
1,230
1,231
1,232
1,233
1,234
1,235
1,236
1,237
1,238
1,239
1,240
1,241
1,242
1,243
1,244
1,245
1,246
1,247
1,248
1,249
1,250
1,251
1,252
1,253
1,254
1,255
1,256
1,257
1,258
1,259
1,260
1,261
1,262
1,263
1,264
1,265
1,266
1,267
1,268
1,269
1,270
1,271
1,272
1,273
1,274
1,275
1,276
1,277
1,278
1,279
1,280
1,281
1,282
1,283
1,284
1,285
1,286
1,287
1,288
1,289
1,290
1,291
1,292
1,293
1,294
1,295
1,296
1,297
1,298
1,299
1,300
1,301
1,302
1,303
1,304
1,305
1,306
1,307
1,308
1,309
1,310
1,311
1,312
1,313
1,314
1,315
1,316
1,317
1,318
1,319
1,320
1,321
1,322
1,323
1,324
1,325
1,326
1,327
1,328
1,329
1,330
1,331
1,332
1,333
1,334
1,335
1,336
1,337
1,338
1,339
1,340
1,341
1,342
1,343
1,344
1,345
1,346
1,347
1,348
1,349
1,350
1,351
1,352
1,353
1,354
1,355
1,356
1,357
1,358
1,359
1,360
1,361
1,362
1,363
1,364
1,365
1,366
1,367
1,368
1,369
1,370
1,371
1,372
1,373
1,374
1,375
1,376
1,377
1,378
1,379
1,380
1,381
1,382
1,383
1,384
1,385
1,386
1,387
1,388
1,389
1,390
1,391
1,392
1,393
1,394
1,395
1,396
1,397
1,398
1,399
1,400
1,401
1,402
1,403
1,404
1,405
1,406
1,407
1,408
1,409
1,410
1,411
1,412
1,413
1,414
1,415
1,416
1,417
1,418
1,419
1,420
1,421
1,422
1,423
1,424
1,425
1,426
1,427
1,428
1,429
1,430
1,431
1,432
1,433
1,434
1,435
1,436
1,437
1,438
1,439
1,440
1,441
1,442
1,443
1,444
1,445
1,446
1,447
1,448
1,449
1,450
1,451
1,452
1,453
1,454
1,455
1,456
1,457
1,458
1,459
1,460
1,461
1,462
1,463
1,464
1,465
1,466
1,467
1,468
1,469
1,470
1,471
1,472
1,473
1,474
1,475
1,476
1,477
1,478
1,479
1,480
1,481
1,482
1,483
1,484

Match case Limit results 1 per page

Send documentation comments to [email protected]

26-14

Cisco MDS 9000 Family CLI Configuration Guide

OL-18084-01, Cisco MDS NX-OS Release 4.x

Chapter 26

Configuring Fibre Channel Routing Services and Protocols

In-Order Delivery

Figure 26-5

, the new path from Switch 1 to Switch 4 is faster. In this scenario, Frame 3 and Frame 4

may be delivered before Frame 1 and Frame 2.

If the in-order guarantee feature is enabled, the frames within the network are treated as follows:

•

Frames in the network are delivered in the order in which they are transmitted.

•

Frames that cannot be delivered in order within the network latency drop period are dropped inside

the network.

About Reordering PortChannel Frames

When a link change occurs in a PortChannel, the frames for the same exchange or the same flow can

switch from one path to another faster path.

Figure 26-6

Link Congestion Delivery

Figure 26-6

, the port of the old path (red dot) is congested. In this scenario, Frame 3 and Frame 4 can

be delivered before Frame 1 and Frame 2.

The in-order delivery feature attempts to minimize the number of frames dropped during PortChannel

link changes when the in-order delivery is enabled by sending a request to the remote switch on the

PortChannel to flush all frames for this PortChannel.

Note

Both switches on the PortChannel must be running Cisco SAN-OS Release 3.0(1) for this IOD

enhancement. For earlier releases, IOD waits for the switch latency period before sending new frames.

When the in-order delivery guarantee feature is enabled and a PortChannel link change occurs, the

frames crossing the PortChannel are treated as follows:

•

Frames using the old path are delivered before new frames are accepted.

•

The new frames are delivered through the new path after the switch latency drop period has elapsed

and all old frames are flushed.

Frames that cannot be delivered in order through the old path within the switch latency drop period are

dropped. See the

“Configuring the Drop Latency Time” section on page 26-17

About Enabling In-Order Delivery

You can enable the in-order delivery feature for a specific VSAN or for the entire switch. By default,

in-order delivery is disabled on switches in the Cisco MDS 9000 Family.

Switch 1

Switch 2

Frame 2

Frame 1

Frame 4

Frame 3

Old path

New path

85475