Home » Cisco Manuals » Hubs & Switches » Cisco MDS-9124 » Manual Viewer

Cisco MDS-9124 Troubleshooting Guide - Page 49

Troubleshooting Installs, Upgrades, and Reboots, Overview

Add to My Manuals
Save this manual to your list of manuals

Page 49 highlights

Send documentation comments to [email protected] 2 C H A P T E R Troubleshooting Installs, Upgrades, and Reboots This chapter describes how to identify and resolve problems that might occur when installing, upgrading, or restarting Cisco MDS 9000 Family products. It includes the following sections: • Overview, page 2-1 • Guidelines, page 2-2 • Disruptive Module Upgrades, page 2-4 • Troubleshooting a Nondisruptive Upgrade on a Fabric Switch, page 2-4 • Troubleshooting a Nondisruptive Upgrade on a Fabric Switch, page 2-4 • Verifying Cisco SAN-OS Software Installations, page 2-6 • Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades, page 2-7 • Troubleshooting Cisco SAN-OS Software System Reboots, page 2-13 • Recovering the Administrator Password, page 2-32 • Miscellaneous Software Image Issues, page 2-32 Overview Each Cisco MDS 9000 switch ships with an operating system (Cisco SAN-OS) that consists of two images-the kickstart image and the system image. There is also a module image if the Storage Services Module (SSM) is present. Installations, upgrades, and reboots are ongoing parts of SAN maintenance activities. It is important to minimize the risk of disrupting ongoing operations when performing these operations in production environments and to know how to recover quickly when something does go wrong. Note For documentation purposes, we use the term upgrade in this document. However, upgrade refers to both upgrading and downgrading your switch, depending on your needs. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-1

Section	Page
Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x	1
Contents	3
New and Changed Information	25
Preface	27
Document Organization	27
Document Conventions	29
Related Documentation	30
Release Notes	30
Compatibility Information	30
Regulatory Compliance and Safety Information	30
Hardware Installation	30
Cisco Fabric Manager	31
Command-Line Interface	31
Troubleshooting and Reference	31
Installation and Configuration Note	31
Obtaining Documentation, Obtaining Support, and Security Guidelines	32
Troubleshooting Overview	33
Overview of the Troubleshooting Process	33
Best Practices	34
Troubleshooting Basics	34
General Steps	34
Gathering Information Using Common Fabric Manager Tools and CLI Commands	35
Verifying Basic Connectivity	36
Verifying SAN Element Registration	37
Fibre Channel End-to-End Connectivity	37
Primary Troubleshooting Flowchart	41
System Messages	42
System Message Text	42
Syslog Server Implementation	42
Implementing Syslog with Fabric Manager	43
Implementing Syslog with the CLI	44
Troubleshooting with Logs	45
Viewing Logs with Fabric Manager	45
Viewing Logs with the CLI	45
Viewing the Log from the Supervisor	46
Contacting Customer Support	47
Troubleshooting Installs, Upgrades, and Reboots	49
Overview	49
Guidelines	50
Guidelines for Installations	50
Guidelines for Upgrading	50
Guidelines for Reboots	52
Disruptive Module Upgrades	52
Troubleshooting a Nondisruptive Upgrade on a Fabric Switch	52
Troubleshooting Fabric Manager Installations	53
Verifying Cisco SAN-OS Software Installations	54
Troubleshooting Cisco SAN-OS Software Upgrades and Downgrades	55
Software Installation Reports an Incompatibility	55
Software Installation Ends with Error	57
Troubleshooting Cisco SAN-OS Software System Reboots	61
Power On or Switch Reboot Hangs	61
Corrupted Bootflash Recovery	62
Recovery Using BIOS Setup for Supervisor-1	64
Recovery from the loader> Prompt on Supervisor-2 Modules	67
Recovery from the loader> Prompt on Supervisor-1 Modules	68
Recovery from the switch(boot)# Prompt	69
Recovery for Switches with Dual Supervisor Modules	70
Recognizing Error States	73
Switch or Process Resets	74
Recoverable System Restarts	75
Unrecoverable System Restarts	79
Recovering the Administrator Password	80
Miscellaneous Software Image Issues	80
All Ports Down Because of System Health Failure	81
Switch Reboots after FCIP Reload	81
FCIP Link Fails to Come Up	81
Cannot Create, Modify, or Delete Admin Role	82
FC IDs Change after Link Reset	82
Switch Displays Wrong User	82
Managing Storage Services Modules	83
SSM Overview	83
Initial Troubleshooting Checklist	85
Common Troubleshooting Tools in Fabric Manager	85
Common Troubleshooting Commands in the CLI	85
SSM Issues	85
SSM Fails to Boot	86
SSM Upgrade Is Disruptive	91
Troubleshooting Hardware	97
Overview	97
SNMP Traps	98
Troubleshooting Startup Issues	98
Troubleshooting Power Supply Issues	99
All Power Supply LEDS Are Off	100
Power Supply Input Ok LED is Red	101
Power Supply Output Failed LED is On	102
Power Supply Fan Ok LED is Red	102
Troubleshooting Fan Issues	104
Fan Is Not Spinning	104
Fan Is Spinning; Fan LED is Red	104
Temperature Threshold Violations	107
Troubleshooting Clock Module Issues	108
Troubleshooting Other Hardware Issues	109
Troubleshooting Supervisor Issues	110
Active Supervisor Reboots	111
Standby Supervisor Not Recognized by Active Supervisor	113
Standby Supervisor Stays in Powered-Up State	115
Troubleshooting Switching and Services Modules	117
Overview of Module Status	117
Module Initialization Overview	118
Troubleshooting Powered-Down Modules	122
Troubleshooting Reloaded Modules	127
Troubleshooting Modules in an Unknown State	130
Troubleshooting Modules Not Detected by the Supervisor	131
Reinitializing a Failed Module Using Fabric Manager	132
Reinitializing a Failed Module Using the CLI	133
Module Resets	134
Troubleshooting Mixed Generation Hardware	135
Overview	135
Port Groups	136
Port Speed Mode	137
Dynamic Bandwidth Management	137
Out-of-Service Interfaces	138
Port Index Availability	138
Combining Modules and Supervisors	141
Initial Troubleshooting Checklist	141
Generation 1 and Generation 2 Issues	141
Module Does Not Come Online	142
Cannot Configure Port in Dedicated Mode	144
Cannot Enable a Port	147
Cannot Upgrade Supervisor System Image	147
Troubleshooting Licensing	149
License Overview	149
Chassis Serial Numbers	150
Grace Period	150
Initial Troubleshooting Checklist	151
Displaying License Information Using Fabric Manager	151
Displaying License Information Using Device Manager	152
Displaying License Information Using Fabric Manager Web Client	152
Displaying License Information Using the CLI	152
Licensing Installation Issues	154
One-Click License Install Fails or Cannot Connect to Licensing Website	154
Serial Number Issues	154
RMA Chassis Errors or License Transfers Between Switches	155
Receiving Grace Period Warnings After License Installation	155
Incorrect Number of Licenses in Use for Multiple Modules	156
Grace Period Alerts	157
Checking in the Fabric Manager Server License From Device Manager	158
License Listed as Missing	159
Troubleshooting Cisco Fabric Services	161
Overview	161
Initial Troubleshooting Checklist	162
Verifying CFS Using Fabric Manager	163
Verifying CFS Using the CLI	163
Merge Failure Troubleshooting	165
Recovering from a Merge Failure with Fabric Manager	166
Recovering from a Merge Failure with the CLI	166
Lock Failure Troubleshooting	167
Resolving Lock Failure Issues Using Fabric Manager	167
Resolving Lock Failure Issues Using the CLI	168
System State Inconsistent and Locks Being Held	169
Distribution Status Verification	169
Verifying Distribution Using Fabric Manager	169
Verifying Distribution Using the CLI	169
CFS Regions Troubleshooting	170
Distribution Failure	170
Regions for Conditional Service	171
Changing Regions	171
Troubleshooting Ports	173
Overview	173
Initial Troubleshooting Checklist	173
Limitations and Restrictions	176
Overview of the FC-MAC Driver and the Port Manager	176
Port Manager Overview	177
Troubleshooting Port States with the Device Manager	178
Troubleshooting Port States from the CLI	182
Using Port Debug Commands	182
Useful Commands at the FC-MAC Level	183
Isolating Port Issues Using the CLI	184
Common Problems with Port Interfaces	185
Port Remains in a Link Failure or Not Connected State	185
Port Remains in Initializing State	188
Unexpected Link Flapping Occurs	193
Port Bounces Between Initializing and Offline States	198
E Port Bounces Remains Isolated After a Zone Merge	200
Port Cycles Through Up and Down States	203
Port Is in ErrDisabled State	203
Troubleshooting Fx Port Failure	204
Troubleshooting N-Port Virtualization	207
Overview	207
Initial Troubleshooting Checklist	208
Limitations and Restrictions	208
Common CLI Commands for NPV	208
Common Problems with NPV	210
Moving the Login of an End Device	210
NPIV Is Not Enabled	211
VSAN Mismatches	211
Core NPV Device Is Not a Switch	212
NPV Core Switch Port Is Down	212
Server Interface is Down	212
Waiting on FLOGI from the Server or Target	213
Waiting on External Link to Come Up	213
Troubleshooting PortChannels and Trunking	215
PortChannel Overview	215
Trunking Overview	216
Initial Troubleshooting Checklist	216
Common Troubleshooting Tools in Fabric Manager	216
Common Troubleshooting Commands in the CLI	217
PortChannel Issues	217
Cannot Configure a PortChannel	217
Newly Added Interface Does Not Come Online In a PortChannel	218
Trunking Issues	218
Cannot Configure Trunking	219
VSAN Traffic Does Not Traverse Trunk	219
Troubleshooting VSANs, Domains, and FSPF	221
Overview	221
Initial Troubleshooting Checklist	221
Common Troubleshooting Tools in Fabric Manager	222
Common Troubleshooting Commands in the CLI	222
VSAN Issues	223
Host Cannot Communicate with Storage	223
E Port Is Isolated in a VSAN	225
Troubleshooting Interop Mode Issues	229
Dynamic Port VSAN Membership Issues	229
Troubleshooting DPVM Using Fabric Manager	230
Troubleshooting DPVM Using the CLI	231
DPVM Configuration Not Available	231
DPVM Database Not Distributed	232
DPVM Autolearn Not Working	232
No Autolearn Entries in Active Database	233
VSAN Membership not Added to Database	233
DPVM Config Database Not Activating	234
Cannot Copy Active to Config DPVM Database	234
Port Suspended or Disabled After DPVM Activation	235
DPVM Merge Failed	235
DPVM Process Terminates	236
Domain Issues	237
Domain ID Conflict Troubleshooting	237
Switch Cannot See Other Switches in a VSAN	238
FC Domain ID Overlap	238
CFS Distribution of Domain ID List Fails	242
Allowed Domain ID List Incorrect After a VSAN Merge	243
Changes to fcdomain Do Not Take Effect	243
FSPF Issues	244
Troubleshooting FSPF	244
Loss of Two-Way Communication	248
Troubleshooting SAN Device Virtualization	255
Overview	255
Initial Troubleshooting Checklist	255
Debugging and Verifying SDV Configuration Using the CLI	256
SDV Limitations and Restrictions	256
SDV Issues	257
SDV Commit Fails	257
SDV Commit Partially Fails	258
Host Cannot Locate Disk	259
SDV Merge Fails When ISL Comes Up	260
Zone Activation Fails in a SDV Zone	260
Troubleshooting IVR	261
Overview	261
Configuration Guidelines	261
Transit VSANs	262
Border Switches	262
Limitations and Restrictions	262
Initial Troubleshooting Checklist	263
Verifying IVR Configuration Using Fabric Manager	263
Verifying IVR Configuration Using the CLI	264
IVR Enhancements by Cisco SAN-OS Release	265
IVR Issues	266
IVR Licensing Issues	266
Cannot Enable IVR	267
IVR Network Address Translation Fails	268
IVR Zone Set Activation Fails	268
Border Switch Fails	270
Traffic Does Not Traverse IVR Path	271
Link Isolated	272
Persistent FC ID for IVR Failed	272
LUN Configuration Failure in IVR Zoning	273
Host Does Not Have Write Access to Storage	273
Locked IVR CFS Session	273
CFS Merge Failed	274
Troubleshooting the IVR Wizard	275
Warning: Not All Switches Are IVR NAT Capable or Are Unmanageable	276
Error: The Following Switches Do Not Have Unique Domain IDs	276
Error: Pending Action/ Pending Commits	277
Error: Fabric Is Changing. Please Retry the Request Later	277
Troubleshooting Zones and Zone Sets	279
Overview	279
Troubleshooting Checklist	279
Troubleshooting Zone Configuration Issues with Fabric Manager	280
Troubleshooting Zone Configuration Issues with the CLI	280
Zone and Zone Set Issues	282
Host Cannot Communicate with Storage	283
Troubleshooting Zone Set Activation	286
Troubleshooting Full Zone Database Synchronization Across Switches	289
Mismatched Default Zone Policy	290
Zone Merge Failure	291
Recovering from Link Isolation	293
Mismatched Active Zone Sets Within the Same VSAN	295
Enhanced Zoning Issues	299
Resolving Enhanced Zoning Lock Issues with Fabric Manager	301
Resolving Enhanced Zoning Lock Issues with the CLI	301
Troubleshooting Distributed Device Alias Services	303
Overview	303
Initial Troubleshooting Checklist	303
Merge Failure Messages	304
Merge Validation Failure Messages	304
Commit Failure Messages	305
Verifying Device Alias Database Status Using the CLI	305
Limitations and Restrictions	305
Merge Failure Issues	306
Resolving Duplicate Device Alias Names	307
Resolving Mapping a pWWN to Different Device Alias Names	308
Resolving Mode Mismatch	308
Resolving Merge Failures in Mixed Fabric	309
Resolving a Validation Failure	310
Resolving Merge in Progress Issues	311
Validation and Commit Failure Issues	313
Resolving Database Conflicts	314
Resolving Application Busy Situations	315
Resolving Database Size Issues	316
Resolving Mode Issues	317
Troubleshooting FICON	319
FICON Overview	319
FICON Port Numbering	320
FICON Configuration Files	323
CUP In-Band Management	324
Fabric Binding	324
FICON Configuration Requirements	324
Initial Troubleshooting Checklist	325
Common Troubleshooting Tools in Fabric Manager or Device Manager	326
Common Troubleshooting Commands in the CLI	326
Port Swapping	326
FICON Issues	327
Cannot Enable FICON	328
Switch ISL Isolated	328
Fabric Manager or Device Manager Cannot Configure FICON	328
Mainframe Cannot Configure FICON	329
Cannot Enable FICON Port	329
Cannot Configure FCIP or PortChannel for FICON	330
FCIP fails for FICON	330
FICON Tape Acceleration Not Working	330
Troubleshooting RADIUS and TACACS+	331
AAA Overview	331
Initial Troubleshooting Checklist	331
Common Troubleshooting Tools in Fabric Manager	332
Common Troubleshooting Commands in the CLI	332
AAA Issues	332
Switch Does Not Communicate with AAA Server	332
User Authentication Fails	338
User Is Not in Any Configured Role	340
User Cannot Access Certain Features	341
Troubleshooting RADIUS and TACACS+ With Cisco ACS	341
Troubleshooting Users and Roles	343
Overview	343
User Accounts	343
Role-Based Authorization	344
Rules and Features for Each Role	345
Initial Troubleshooting Checklist	346
Common Troubleshooting Tools in Fabric Manager	346
Common Troubleshooting Commands in the CLI	346
User and Role Issues	346
User Cannot Log into Switch	347
User Cannot Create Roles	349
User Cannot Create Other Users With Fabric Manager or Device Manager	349
User Cannot Access Certain Features	350
User Has Too Much Access	352
User Cannot Configure Some VSANs	352
User Cannot Configure E Ports	353
Unexpected User Displayed in Logs	354
Troubleshooting Users and Roles with Cisco ACS	354
Troubleshooting FC-SP, Port Security, and Fabric Binding	355
FC-SP Overview	355
Port Security Overview	356
Fabric Binding Overview	356
Initial Troubleshooting Checklist	356
Common Troubleshooting Tools in Fabric Manager	357
Common Troubleshooting Commands in the CLI	357
FC-SP Issues	358
Switch or Host Blocked from Fabric	358
Authentication Fails When Using Cisco ACS	361
Port Security Issues	361
Device Does Not Log into a Switch When AutoLearn Is Disabled	362
Device Does Not Log into a Switch When Autolearn Is Enabled	362
Cannot Activate Port Security	366
Unauthorized Device Gains Access to Fabric	366
Port Security Settings Lost After Reboot	367
Merge Fails	368
Fabric Binding Issues	369
Switch Cannot Attach to the Fabric	370
Cannot Activate Fabric Binding	372
Unauthorized Switch Gains Access to Fabric	373
Fabric Binding Settings Lost After Reboot	373
Troubleshooting IP Storage Services	375
Overview	375
iSCSI Restrictions	376
iSLB Restrictions	376
Initial Troubleshooting Checklist	377
Common Troubleshooting Tools in Fabric Manager	377
Common Troubleshooting Commands in the CLI	377
IP Issues	379
Verifying Basic Connectivity	379
Verification of Switch Connectivity	381
Verification of Static IP Routing	382
Cannot Assign IP Address to an Interface	383
FCIP Issues	383
One-to-One FCIP Tunnel Creation and Monitoring	384
One-to-Three FCIP Tunnel Creation and Monitoring	394
FCIP Profile Misconfiguration Examples	395
FCIP Interface Misconfiguration Examples	398
FCIP Special Frame Tunnel Creation and Monitoring	404
Special Frame Misconfiguration Example	407
Troubleshooting FCIP Link Flaps	408
Troubleshooting FCIP ISL Link Failures	408
Troubleshooting FCIP and Compression	409
iSCSI Issues	409
Troubleshooting iSCSI Authentication	409
Displaying iSCSI Authentication Using Fabric Manager	410
Displaying iSCSI Authentication Using the CLI	410
Troubleshooting User Name and Password Configuration	411
RADIUS Configuration Troubleshooting	412
Troubleshooting RADIUS Routing Configuration	414
Troubleshooting Dynamic iSCSI Configuration	415
iSCSI TCP Performance Issues	422
CLI Commands Used to Access Performance Data	423
Understanding TCP Parameters for iSCSI	423
Lab Setup	424
Configuring from the Bottom Switch Using the CLI	424
TCP Parameter Changes	428
iSLB Issues	432
iSLB Configuration Not Distributed to All Switches in the Fabric	432
iSCSI Initiator and Virtual Target Configuration Not Distributed	433
iSLB Configuration, Commit, or Merge Failed-”VSAN ID is Not Yet Configured”	433
iSLB Configuration, Commit, or Merge Failed-”Failed to Allocate WWN”	434
iSLB Configuration, Commit, or Merge Failed-”Duplicate WWN Found as...”	434
iSLB Configuration, Commit, or Merge Failed-”Duplicate Node Name”	434
iSLB Configuration Failed-”Pending iSLB CFS Config Has Reached Its Limit...”	435
iSCSI Disable Failed-”Cannot Disable Iscsi - Large Iscsi Config Present...”	435
iSLB Commit Timeout	435
Session Down-”pWWN in Use At Remote Switch”	436
Redirected Session Does Not Come Up	436
iSLB Zones Not Present in Active Zone Set	437
Traffic Description After iSLB Commit or Activation of Zone Set	437
VRRP Master Overutilized	438
iSLB Zone Set Activation Failed	438
iSLB CFS Commit Fails	439
Resolving an iSLB Merge Failure	439
Troubleshooting IP Access Lists	441
Overview	441
Protocol Information	441
Address Information	442
Port Information	442
ICMP Information	443
ToS Information	443
Initial Troubleshooting Checklist	444
Common Troubleshooting Tools in Fabric Manager	444
Common Troubleshooting Commands in the CLI	444
IP-ACL Issues	444
All Packets Are Blocked	445
No Packets Are Blocked	447
PortChannel Not Working with ACL	448
Cannot Remotely Connect to Switch	448
Troubleshooting IPsec	449
Overview	449
IPsec Compatibility	449
Supported IPsec and IKE Algorithms for Microsoft Windows and Linux Platforms	450
IKE Allowed Transforms	451
IPsec Allowed Transforms	452
Initial Troubleshooting Checklist	452
Common Troubleshooting Tools in Fabric Manager	452
Common Troubleshooting Commands in the CLI	453
IPsec Issues	453
Verifying IKE Configuration Compatibility	454
Verifying IPsec Configuration Compatibility Using Fabric Manager	454
Verifying IPsec Configuration Compatibility Using the CLI	455
Verifying Security Policy Databases Compatibility	456
Verifying Interface Status Using Fabric Manager	457
Verifying Interface Status Using the CLI	457
Verifying Security Associations	460
Security Associations Do Not Re-Key	463
Clearing Security Associations	463
Debugging the IPsec Process	463
Debugging the IKE Process	463
Obtaining Statistics from the IPsec Process	463
Troubleshooting SANTap	465
Overview	465
Definitions	466
Limitations	466
Interface Restrictions	467
Initial Troubleshooting Checklist	467
Common Troubleshooting Tools in Fabric Manager	468
Common Troubleshooting Commands in the CLI	468
Messages, Logs and Databases	470
SANTap Issues	470
Host Login Problems	471
ITL Problems	471
Common Mismatch Problems	471
Troubleshooting Digital Certificates	475
Overview	475
Digital Certificates	475
Certificate Authorities	475
RSA Key Pairs and Identity Certificates	476
Peer Certificate Verification	476
CRLs and OCSP Support	476
Import and Export Support for Certificates and Associated Key Pairs	476
PKI Enrollment Support	476
Maximum Limits	477
Initial Troubleshooting Checklist	477
Common Troubleshooting Tools in Fabric Manager	477
Common Troubleshooting Commands in the CLI	478
Digital Certificate Issues	478
CA Will Not Generate Identity Certificate	478