Cisco MDS-9124 Troubleshooting Guide
Cisco MDS-9124 Manual
 |
View all Cisco MDS-9124 manuals
Add to My Manuals
Save this manual to your list of manuals |
Cisco MDS-9124 manual content summary:
- Cisco MDS-9124 | Troubleshooting Guide - Page 1
Send documentation comments to [email protected] Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Cisco MDS SAN-OS for Release 3.3(1) May 2008 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553- - Cisco MDS-9124 | Troubleshooting Guide - Page 2
, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x © 2009 Cisco Systems, Inc. All rights reserved. - Cisco MDS-9124 | Troubleshooting Guide - Page 3
Fibre Channel End-to-End Connectivity 1-5 Fabric Issues 1-5 Port Issues 1-6 Primary Troubleshooting Flowchart 1-9 System Messages 1-10 System Message Text 1-10 Syslog Server Implementation 1-10 Implementing Syslog with Fabric Manager 1-11 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iii - Cisco MDS-9124 | Troubleshooting Guide - Page 4
With Corrupted Bootflash 2-23 Recovering Both Supervisor Modules With Corrupted Bootflash 2-24 Recognizing Error States 2-25 Switch or Process Resets 2-26 Recoverable System Restarts 2-27 Unrecoverable System Restarts 2-31 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x iv OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 5
Troubleshooting the Power Supplies 4-7 Troubleshooting Fan Issues 4-8 Fan Is Not Spinning 4-8 Fan Is Spinning; Fan LED is Red 4-8 Troubleshooting a Fan Failure Using Device Manager 4-9 Troubleshooting a Fan Failure Using the CLI 4-10 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 6
Failed Module Using the CLI 4-37 Module Resets 4-38 5 C H A P T E R Troubleshooting Mixed Generation Hardware 5-1 Overview 5-1 Port Groups 5-2 Port Speed Mode 5-3 Dynamic Bandwidth Management 5-3 Out-of-Service Interfaces 5-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x vi OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 7
Manager 6-10 License Listed as Missing 6-11 7 C H A P T E R Troubleshooting Cisco Fabric Services 7-1 Overview 7-1 Initial Troubleshooting Checklist 7-2 Verifying CFS Using Fabric Manager 7-3 Verifying CFS Using the CLI 7-3 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x vii - Cisco MDS-9124 | Troubleshooting Guide - Page 8
Remains in a Link Failure or Not Connected State 8-13 Troubleshooting Port Problems 8-15 Port Remains in Initializing State 8-16 Troubleshooting Port Registration Issues Using the CLI 8-17 Unexpected Link Flapping Occurs 8-21 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x viii OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 9
Troubleshooting Commands in the CLI 10-3 PortChannel Issues 10-3 Cannot Configure a PortChannel 10-3 Newly Added Interface Does Not Come Online In a PortChannel 10-4 Configuring Port Channel Modes Using Fabric Manager 10-4 Trunking Issues 10-4 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 10
cisco.com Cannot Configure Trunking 10-5 VSAN Traffic Does Not Traverse Trunk 10-5 11 C H A P T E R Troubleshooting VSANs, Domains, and FSPF 11-1 Overview 11-1 Initial Troubleshooting Checklist 11-1 Common Troubleshooting Issues 11-17 Domain ID Conflict Troubleshooting 11-17 Switch Cannot See - Cisco MDS-9124 | Troubleshooting Guide - Page 11
OL-9285-05 Troubleshooting IVR 13-1 Overview 13-1 Configuration Guidelines 13-1 Transit VSANs 13-2 Border Switches 13-2 Limitations and Restrictions 13-2 Initial Troubleshooting Checklist 13-3 Verifying IVR Configuration Using Fabric Manager 13-3 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 12
Write Access to Storage 13-13 Locked IVR CFS Session 13-13 CFS Merge Failed 13-14 Troubleshooting the IVR Wizard 13-15 Warning: Not All Switches Are IVR NAT Capable or Are Unmanageable Default Zone Policies Using the CLI 14-13 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xii OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 13
Size Issues 15-14 Resolving Mode Issues 15-15 16 C H A P T E R Troubleshooting FICON 16-1 FICON Overview 16-1 FICON Port Numbering 16-2 Default FICON Port Numbering Scheme 16-2 Reserved FICON Port Numbering Scheme 16-4 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiii - Cisco MDS-9124 | Troubleshooting Guide - Page 14
-9 Verifying RADIUS Server Groups Using the CLI 17-9 Verifying TACACS+ Server Groups Using Fabric Manager 17-9 Verifying TACACS+ Server Groups Using the CLI 17-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xiv OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 15
Fabric Binding Overview 19-2 Initial Troubleshooting Checklist 19-2 Common Troubleshooting Tools in Fabric Manager 19-3 Common Troubleshooting Commands in the CLI 19-3 FC-SP Issues 19-4 Switch or Host Blocked from Fabric 19-4 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xv - Cisco MDS-9124 | Troubleshooting Guide - Page 16
E R Troubleshooting IP Storage Services 20-1 Overview 20-1 iSCSI Restrictions 20-2 iSLB Restrictions 20-2 Initial Troubleshooting Checklist 20-3 Common Troubleshooting Tools in Fabric Manager 20-3 Common Troubleshooting Commands in the CLI 20-3 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 17
Contents Send documentation comments to [email protected] IP Issues 20-5 Verifying Basic Connectivity 20-5 Verifying Basic Connectivity Using Device Manager 20-5 an FCIP Tunnel with Special Frame Using the CLI 20-31 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xvii - Cisco MDS-9124 | Troubleshooting Guide - Page 18
the Bottom Switch Using the CLI 20-50 Verifying Connectivity Between Client and IPS iSCSI Service 20-51 TCP Parameter Changes 20-54 Displaying the Gigabit Ethernet Interface 20-54 Verifying Iscsi Config Present..." 20-61 xviii Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 19
Transforms 22-4 Initial Troubleshooting Checklist 22-4 Common Troubleshooting Tools in Fabric Manager 22-4 Common Troubleshooting Commands in the CLI 22-5 IPsec Issues 22-5 Verifying IKE Configuration Compatibility 22-6 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xix - Cisco MDS-9124 | Troubleshooting Guide - Page 20
and Associated Key Pairs 24-2 PKI Enrollment Support 24-2 Maximum Limits 24-3 Initial Troubleshooting Checklist 24-3 Common Troubleshooting Tools in Fabric Manager 24-3 Common Troubleshooting Commands in the CLI 24-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xx OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 21
Home Alerts 25-8 Not Receiving Syslog-based Call Home Alerts 25-8 Periodic Inventory Notification Does Not Reflect Current Inventory 25-9 26 C H A P T E R Troubleshooting Fabric Manager 26-1 Overview 26-1 Guidelines 26-2 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxi - Cisco MDS-9124 | Troubleshooting Guide - Page 22
Method 26-12 Troubleshooting Performance Manager 26-12 Performance Manager Generates Java Error 26-13 Performance Manager Not Working 26-13 Troubleshooting Device Manager 26-13 Wrong Version of Device Manager is Launched 26-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxii OL - Cisco MDS-9124 | Troubleshooting Guide - Page 23
the show processes cpu CLI Command B-9 Using the show system resource CLI Command B-10 Using On-Board Failure Logging B- Service B-18 SCSI Target Discovery B-19 SNMP and RMON Support B-19 Using RADIUS B-21 Using Syslog B-21 Logging Levels B-22 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 24
27 Using Other Troubleshooting Products B-27 Fibre Channel Testers B-28 Fibre Channel Protocol Analyzers B-28 Using Host Diagnostic Tools B-28 C A P P E N D I X INDEX Configuration Limits for Cisco MDS SAN-OS Release 3.x C-1 xxiv Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285 - Cisco MDS-9124 | Troubleshooting Guide - Page 25
for the Cisco MDS SAN-OS Release 3.x software. The Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x is updated to address each new and changed guideline in the Cisco MDS SAN-OS Release 3.x software. The latest version of this document is available at the following Cisco Systems website: http - Cisco MDS-9124 | Troubleshooting Guide - Page 26
. 3.1(2) Chapter 15, "Troubleshooting Distributed Device Alias Services" Added IP Network Simulator as a troubleshooting 3.1(2) tool. Appendix B, "Troubleshooting Tools and Methodology" Added information about troubleshooting 3.1(2) nondisruptive upgrades on the Cisco MDS 9124 Fabric Switch - Cisco MDS-9124 | Troubleshooting Guide - Page 27
used to troubleshoot Cisco Fabric Services (CFS) problems. Describes how to identify and resolve problems that might occur when using port interfaces. Describes how to identify and resolve problems that might occur when using NPV. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 28
. Describes procedures to troubleshoot Call Home. Describes procedures used to troubleshoot Fabric Manager. Describes the steps to perform before calling for technical support with any Cisco MDS 9000 Family product. xxviii Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285 - Cisco MDS-9124 | Troubleshooting Guide - Page 29
are in angle brackets. [ ] Default responses to system prompts are in square brackets. !, # An manual. Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 30
Quick Start Guide • Cisco MDS 9500 Series Hardware Installation Guide • Cisco MDS 9200 Series Hardware Installation Guide • Cisco MDS 9100 Series Hardware Installation Guide • Cisco MDS 9020 Fabric Switch Hardware Installation Guide Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxx - Cisco MDS-9124 | Troubleshooting Guide - Page 31
Guide Troubleshooting and Reference • Cisco MDS 9000 Family Troubleshooting Guide • Cisco MDS 9000 Family MIB Quick Reference • Cisco MDS 9020 Fabric Switch MIB Quick Reference • Cisco MDS 9000 Family SMI-S Programming Reference • Cisco MDS 9000 Family System Messages Reference • Cisco MDS - Cisco MDS-9124 | Troubleshooting Guide - Page 32
aliases and general Cisco documents, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html xxxii Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL - Cisco MDS-9124 | Troubleshooting Guide - Page 33
the possible problems, you need to use a variety of tools and understand the overall storage environment. For this reason, this guide describes a number of general troubleshooting tools in Appendix B, "Troubleshooting Tools and Methodology," including those that are specific to the Cisco MDS 9000 - Cisco MDS-9124 | Troubleshooting Guide - Page 34
to determine the status of your installation: • Is this a newly installed system or an existing installation? (It could be a new SAN, host, problem (too slow, too high latency, excessively long response time) or did the problem show up recently? Cisco MDS 9000 Family Troubleshooting Guide, - Cisco MDS-9124 | Troubleshooting Guide - Page 35
troubleshoot problems within your fabric. These tools and commands are a subset of what you may use to troubleshoot your specific problem. Each chapter in this guide . • Cisco SAN-OS version-In Device Manager, choose Physical > System. • Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 36
Basics Chapter 1 Troubleshooting Overview Send documentation comments to [email protected] • View traffic statistics-In Device Manager, click the Device used, is the host's pWWN listed in the LUN masking database? Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 37
system logs to determine if the subsystem pWWN or FC ID is listed as a device. This can validate that FSPF is working correctly. Fabric Issues Answering the following questions will help to determine the status of the fabric configuration: OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 38
timeout values. • Yellow box: In Device Manager, a port has been selected. • Gray box: The port is administratively disabled. • Black box: An SFP is not present. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 39
connected switch • Speed • Frames transmitted and received • Percentage utilization for the CPU, dynamic memory, and Flash memory Figure 1-2 Device Manager: Summary View OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 40
of service for N port connections. • ELP-View pWWN, nWWN, BB_credits, and supported classes of service for supported switching modules and line cards. Use the Device View to view an individual module and then right-click to select from the following tasks: Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 41
oversubscription is supported on 24-port and 48-port 4-Gbps Fibre Channel switching modules, the 32-port 2-Gbps Fibre Channel switching module, the Cisco MDS 9120 20-port 1/2-Gbps Fibre Channel module, and the Cisco MDS 9140 40-port 1/2-Gbps Fibre Channel module. Primary Troubleshooting Flowchart - Cisco MDS-9124 | Troubleshooting Guide - Page 42
example will demonstrate how to configure a Cisco MDS switch to utilize the syslog facility on a Solaris platform. Although a Solaris host is being used, syslog configuration on all UNIX and Linux systems is very similar. 1-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 43
those messages. These messages can be categorized into four classes: • Hardware-Line card or power supply problems • Link incidents-FICON port condition changes • Accounting-User change events • Events-All other events OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 44
11:07:51 [172.22.36.142.2.2] : 2004 Sep 17 11:17:39 pacific: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring console from pts/0 (dhcp-171-71-49-125.cisco.com) 1-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 45
to [email protected] Troubleshooting with Logs Cisco SAN-OS generates many types of system messages on the switch and sends them to a syslog server. These messages can be viewed using Fabric Manager or the CLI to determine what events may have led up to the current problem condition you - Cisco MDS-9124 | Troubleshooting Guide - Page 46
Use the show logging CLI command to view the logs on the supervisor. Viewing NVRAM logs System messages that are priority 0, 1, or 2 are logged into NVRAM on the supervisor module. of module 14 (serial: JAB092501FC) failed 1-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 47
documentation comments to [email protected] Contacting Customer Support If you are unable to solve a problem after using the troubleshooting suggestions in this guide, contact a customer service representative for assistance and further instructions. Before you call, have the following - Cisco MDS-9124 | Troubleshooting Guide - Page 48
Contacting Customer Support Chapter 1 Troubleshooting Overview Send documentation comments to [email protected] 1-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 49
Upgrades and Downgrades, page 2-7 • Troubleshooting Cisco SAN-OS Software System Reboots, page 2-13 • Recovering the Administrator Password, page 2-32 • Miscellaneous Software Image Issues, page 2-32 Overview Each Cisco MDS 9000 switch ships with an operating system (Cisco SAN-OS) that consists of - Cisco MDS-9124 | Troubleshooting Guide - Page 50
the directories of the MDS file system. Guidelines for Upgrading Not Cisco SAN-OS Release Notes are available at the following website: http://cisco.com/en/US/products/ps5989/prod_release_notes_list.html • Ensure that an FTP or TFTP server is available. Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 51
integrity is automatically checked, including the running kickstart and system images. - The command performs a platform validity check ends. In such cases, you can verify the problem on the affected switching module and upgrade the other Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 52
/2, MSM-18/4 module, or the IP Storage (IPS) services modules are disruptive. These modules use a rolling upgrade install mechanism system prompt to enter the show all failure-reason command. Install has failed. Return code 0x401E0066 (request timed out). Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 53
show tech-support command output and the console output from the installation, if available. Troubleshooting Fabric Manager Installations This section describes possible problems and dialog box, and restart Device Manager. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 54
Panel > Administrative Tools > Services to verify that Fabric Manager problem remains, then remove the application using the Cisco MDS 9000/Uninstall program, then reinstall Fabric Manager. Verifying Cisco "system" version Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 55
or the install all CLI command. Correct any problems and retry the installation. See the "Diagnosing supported by the system image; as a result, some resources might become unavailable after an install. Do you wish to continue? (y/ n) [y]: n OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 56
tunnel feature is not supported in the new image. The RSPAN feature uses Fibre Channel tunnels. 2) Feature Index : 119 , Capability : CAP_FEATURE_FC_TUNNEL_CFG Description : fc-tunnel is enabled Capability requirement : STRICT Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 57
See the "Installing SAN-OS Software Using Fabric Manager" section on page 2-10 or the "Installing Cisco SAN-OS Software from the CLI" section on page 2-11. Or, use the install module CLI command to upgrade the failed module. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 58
image for each switch to proceed. Check the active (and standby, if applicable) bootflash: file system on each switch to see if there is enough space for the new images. You can see files from the local host to the switch. 2-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 59
to stop the installation and contact the next level of support. ca-9506# install all system scp://testuser@dino/tftpboot/rel/qa/2_1_1a/final/m95 00- Extracting "system" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 60
upgrade is not supported Hitless upgrade is not supported Images will be bios v1.1.0(10/24/03) v1.1.0(10/24/03) 5 loader 1.2(2) 1.2(2) 6 system 2.0(2b) 2.1(1a) 6 kickstart 2.0(2b) 2.1(1a) 6 bios v1.1.0(10/24 12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 61
modules (supervisor and switching) are upgraded. Troubleshooting Cisco SAN-OS Software System Reboots This section lists possible problems and solutions for software reboots and includes the -1" section on page 2-16. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 62
Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to [email protected] Table 2-5 Problem to recover a corrupted bootflash. 2-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 63
No Recovery BIOS configuration Hanging state loader> prompt Hanging state Switch (boot)# prompt 3 = Kickstart image Power on and Ctrl-C 3 = Kickstart image Power on and Esc 4 = System image 79950 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 64
Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to mdsfeedback-doc@cisco. not work if your terminal is not configured properly.) 2-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 65
Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to [email protected] Step 4 Press the Tab key to select the Basic CMOS Configuration. You see the System as shown on your TFTP server. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 66
Step 14 Enter the init system command at the switch(boot)# prompt, and press Enter to reformat the file system. switch(boot)# init system Note The init system command also installs a (boot)# Prompt" section on page 2-21. 2-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 67
is not supported, so you system Caution Be sure that you have made a backup of the configuration files before you issue this command. Step 6 Follow the procedure specified in the "Recovery from the switch(boot)# Prompt" section on page 2-21. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 68
want the command to appear. • The ICMP stack is not supported, so you cannot ping the TFTP server. • The TFTP in that list. To recover a corrupted kickstart image (system error state) for a switch with a single supervisor module Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 69
system and kickstart image files are copied to your bootflash: file system. switch(boot)# dir bootflash: 46080 Nov 11 21:48:55 2008 14753280 Nov 11 21:39:49 2008 78473925 Nov 11 21:37:12 2008 lost+found/ kickstart-image1 system-image1 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 70
Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to [email protected] Step 9 Usage for bootflash://sup-local 107465728 bytes used 78619648 bytes free 186085376 bytes total Load the system image from - Cisco MDS-9124 | Troubleshooting Guide - Page 71
module to avoid corrupting the internal bootflash:. After the init system command completes on the standby supervisor module, issue the system no standby manual-boot command in EXEC mode on the active supervisor module. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-23 - Cisco MDS-9124 | Troubleshooting Guide - Page 72
Troubleshooting Cisco SAN-OS Software System Reboots Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Send documentation comments to mdsfeedback-doc@cisco = "2.1(2 Image verification OK Starting kernel... 2-24 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 73
switch(boot)# prompt indicates that you have a usable Kickstart image. Issue the init-system command to repartition and format the bootflash. Perform the procedure specified in the "Recovery Powered On and Ctrl-C Is Entered OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-25 - Cisco MDS-9124 | Troubleshooting Guide - Page 74
error occurred on the system. Cisco SAN-OS cannot recover automatically from the problem. See the "Unrecoverable System Restarts" section on page 2-31 to determine the cause. A clock module failed. Verify that a clock module failed. See the "Troubleshooting Clock Module Issues" section - Cisco MDS-9124 | Troubleshooting Guide - Page 75
, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to [email protected] Recoverable System Restarts Every process restart generates a syslog message and a Call Home event. Even if the event does not affect service, you should identify - Cisco MDS-9124 | Troubleshooting Guide - Page 76
information about a specific process that has restarted: switch# show processes log pid 898 Service: idehsd Description: ide hotswap handler Daemon Started at Mon Sep 16 14:56:04 7FFFFD14 2AC2C581 0804E6BC 7FFFFCA8 2-28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 77
one-time occurrence, compare the length of time that the system has been up with the timestamp of each restart. Enter in the log directory: switch# show pro log pid 1473 Service: ips Description: IPS Manager Started at Tue Jan 8 17: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-29 - Cisco MDS-9124 | Troubleshooting Guide - Page 78
system cores manually support representative and asking the representative to review your core dump. See also the "Troubleshooting Supervisor Issues" section on page 4-14 or the "Troubleshooting Switching and Services Modules" section on page 4-21. 2-30 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 79
Example 2-2 show system reason-reset Command Output switch# show system reset-reason module Service: Version:2.1(2) 4) At 255842 usecs after Wed Jan 19 00:07:49 2005 Reason: Reset Requested by CLI command reload Service: Version: 2.1(2) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 80
Problem You forgot the administrator password for accessing a Cisco MDS 9000 Family switch. Solution You can recover the password using a local console connection. For the latest instructions on password recovery, refer to the Cisco MDS 9000 CLI Family Configuration Guide of System Health Failure - Cisco MDS-9124 | Troubleshooting Guide - Page 81
upgrade from Cisco MDS SAN-OS Release 2.0(1b) to Release 2.0(3) and following the configuration of a new FCIP link. Reload the MPS-14/2 module using the reload module module-number command, where module-number is a specific module. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 82
. The user shown after the nondisruptive upgrade is different from the user shown when you issue the show user-account command. Recreate the user. 2-34 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 83
chapter includes the following sections: • SSM Overview, page 3-1 • Initial Troubleshooting Checklist, page 3-3 • SSM Issues, page 3-3 SSM Overview The 32-port Fibre Channel Storage Services Module (SSM) for the Cisco MDS 9000 Family supports up to 32 Fibre Channel ports and provides distributed - Cisco MDS-9124 | Troubleshooting Guide - Page 84
Layer 3 Intelligent Storage Services on your SSM, we recommend that you shut down these services before upgrading the SSI boot image. You can use dual fabric configuration to minimize the impact of shutting down Layer 3 services. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-2 OL-9285 - Cisco MDS-9124 | Troubleshooting Guide - Page 85
epld • show ssm provisioning SSM Issues This section describes troubleshooting issues for the SSM and SSI images, and it includes the following topics: • SSM Fails to Boot, page 3-4 • SSM Upgrade Is Disruptive, page 3-9 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 86
SSM initially operates in Fibre Channel switching mode by default. Note If you downgrade to a Cisco MDS SAN-OS release that does not support the SSM, you must power down the module. The boot variables for the SSM are lost. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 87
58:44 1980 lost+found/ 27602159 Jul 30 23:05:16 1980 system-image1 12447232 Aug 05 15:08:30 1980 kickstart-image2 28364853 Aug 05 15:11:57 1980 system-image2 4004128 Sep 26 13:43:02 2005 m9000-ek9-ssi-mz.2.1.2.bin ... OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 88
system-image1 12447232 Aug 05 15:08:30 1980 kickstart-image2 28364853 Aug 05 15:11:57 1980 system- Services, a warning displays at the command prompt indicating that the operation will disrupt traffic and you are then asked if you wish to continue. Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 89
Services Module DS-X9032-SSM ok 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby ... Note the slot number for later reference. Verify the Cisco MDS # show boot sup-1 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 90
Type Model Status 4 32 Storage Services Module DS-X9032-SSM ok 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby Mod Sw Hw the values for later reference. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 91
Notes for Cisco MDS 9000 EPLD Images to verify whether or not the EPLD has changed for the Cisco SAN-OS image version being used. Caution Do not insert or remove any modules while an EPLD upgrade or downgrade is in progress. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 92
Mar 21 15:34:46 2005 m9500-sf1ek9-mz.2.1.1.bin 48036239 Apr 06 16:45:41 2005 m9500-sf1ek9-mz.2.1.1a.bin Usage for bootflash://sup-local 141066240 bytes used 43493376 bytes free 184559616 bytes total 3-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 93
m9500-sf1ek9-mz.2.1.1a.bin Usage for bootflash://sup-local 141066240 bytes used 43493376 bytes free 184559616 Services Module DS-X9032-SSM ok 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 94
3 Managing Storage Services Modules Send documentation comments to [email protected] Usage for -epld-2.1.2.img bootflash:m9000-epld-2.1.2.img Note The system will automatically synchronize the ELPD image to the standby 12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 95
comments to [email protected] Note When you upgrade the EPLD module on Cisco MDS 9100 Series switches, you receive the following message: Data traffic on the switch will stop now!! Do you want to continue (y/n)? OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 3-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 96
SSM Issues Chapter 3 Managing Storage Services Modules Send documentation comments to [email protected] 3-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 97
Troubleshooting Supervisor Issues, page 4-14 • Troubleshooting Switching and Services Modules, page 4-21 Overview The key to success when troubleshooting the system hardware is to isolate the problem to a specific system . You OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 98
if you have trouble with the system software. Status LEDs on the supervisor module indicate whether or not the supervisor module can initialize a switching module. If you have a redundant supervisor module, refer to the following website for the latest Cisco MDS 9000 Family configuration guides for - Cisco MDS-9124 | Troubleshooting Guide - Page 99
supply problems and includes the following topics: • All Power Supply LEDS Are Off, page 4-4 • Power Supply Input Ok LED is Red, page 4-5 • Power Supply Output Failed LED is On, page 4-6 • Power Supply Fan Ok LED is Red, page 4-6 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 100
supply [chars]. This reduces the redundant power available to the system and can cause service disruptions (Serial No. [chars]). Explanation Detected a new supply has been removed. Recommended Action No action is required. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 101
power sources to each redundant power supply in a Cisco MDS 9500 Series director. Power supply is not operational. Troubleshoot the power supplies. See the "Troubleshooting the Power Supplies" section on page 4-7. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 102
Troubleshooting the Power Supplies" section on page 4-7. Power Supply Fan Ok LED is Red Symptom Power supply Fan Ok LED is red. The following system . Troubleshoot the power supplies. See the "Troubleshooting the Power Supplies" section on page 4-7. Cisco MDS 9000 Family Troubleshooting Guide, - Cisco MDS-9124 | Troubleshooting Guide - Page 103
4 DS-X9308-SMIP 210.00 5.00 210.00 5.00 5 DS-X9530-SF1-K9 220.08 5.24 220.08 5.24 Status ---------powered-up powered-up powered-up Power Usage Summary Power Supply redundancy mode: Total Power Capacity redundant 1019.34 W OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 104
your customer support representative. Troubleshooting Fan Issues This section describes fan failure problems and system. Fan module has failed. Troubleshoot the Fan Module. See the "Troubleshooting a Fan Failure Using the CLI" section on page 4-10. Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 105
you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you purchased support directly from Cisco, contact Cisco Technical Support at this URL: http://www.cisco.com/warp/public/687/Directory/DirTAC.shtm OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 106
If you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you purchased support directly from Cisco, contact Cisco Technical Support at this URL: http://www.cisco.com/warp/public/687/Directory/DirTAC.shtm 4-10 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 107
or a major threshold violation on an outlet sensor results in the following system message: Error Message PLATFORM-0-MOD_TEMPMAJALRM: Module [dec] reported major temperature alarm. command to collect more information. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 108
. Explanation Chassis clock source has failed and system will be reset. System will automatically start using the redundant clock module. Recommended Action Replace the failed clock module during the next maintenance window. 4-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 109
Possible failed module exception information --- exception instance 1 ---- device id: 12 device errorcode: 0x80000080 system time: (1127843531 ticks) Tue Sep 27 17:52:11 2005 error type: FATAL error Error Type field OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 110
support representative when debugging the problems. • Some interrupt counts may show up under UP-XBAR and DOWN-XBAR ASICs, when one of Supervisors is pulled out or restarted. Troubleshooting nvram • show system reset-reason • show Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 111
active supervisor. Display the system message log on the Service "xbar" (PID 2349) hasn't caught signal 9 (no core). 2005 Sep 27 18:58:06 172.20.150.204 %SYSMGR-3-SERVICE_CRASHED: Service "xbar" (PID 2352) hasn't caught signal 9 (no core). OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 112
exception information --- exception instance 1 ---- device id: 12 device errorcode: 0x80000020 system time: (1127917068 ticks) Wed Sep 28 14:17:48 2005 error type: 18:00 172.20.150.204 %CALLHOME-2-EVENT: SUP_FAILURE 4-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 113
on page 4-17 to verify the problem. Observe the boot process to verify 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw cisco.com (172.22.22.55). Escape character is '^]'. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 114
Chapter 4 Troubleshooting Hardware Send documentation comments to [email protected] Step 3 Step 4 MDS Switch login: admin Password: Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2005, Cisco Systems, Inc. All - Cisco MDS-9124 | Troubleshooting Guide - Page 115
Troubleshooting Hardware Troubleshooting Supervisor Issues Send documentation comments to [email protected] Step 5 dpvm dstats epld_upgrade epp 0 N/A 1 0 0 N/A 1 1 Use the show system internal sysmgr service -9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-19 - Cisco MDS-9124 | Troubleshooting Guide - Page 116
Model Status 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 powered-up 8 8 IP Storage Services Module powered-dn Mod Sw Hw World is used to describe the problem and the workaround. 4-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 117
boot parameters. In either case, the problem was mismatched code on the active and show module 8 Mod Ports Module-Type Model Status 8 8 IP Storage Services Module DS-X9308-SMIP ok Mod Sw Hw World-Wide-Name(s) (WWN) 8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-21 - Cisco MDS-9124 | Troubleshooting Guide - Page 118
Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to mdsfeedback-doc@cisco the module as required by your customer support representative. Once all necessary data is system reset-reason module 4-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 119
Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to mdsfeedback-doc@cisco the code, the supervisor generates the following system message: Error Message MODULE-2-MOD_DNLD_FAIL: Image Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-23 - Cisco MDS-9124 | Troubleshooting Guide - Page 120
Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to [email protected] In addition, the module generates a system determine which component reported the problem. Online and Operational After Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 121
present in the system messages or in Troubleshooting Module Issues To isolate a module problem, follow these steps: Step 1 Verify that all Status LEDs are green. If any status LED is red or off, the module might have shifted out of its slot. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 122
problem with the startup, gather the information listed under Appendix A, "Before Contacting Technical Support" and contact your technical support See the "Troubleshooting Cisco SAN-OS Software System Reboots" section on page 2-13. 4-26 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x - Cisco MDS-9124 | Troubleshooting Guide - Page 123
Chapter 4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to [email protected] Error Message PLATFORM-2- module has been powered down due to errors. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-27 - Cisco MDS-9124 | Troubleshooting Guide - Page 124
problems the show system reset-reason module DS-X9530-SF1-K9 ha-standby 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 8 8 IP Storage Services Module powered-dn Mod Sw Hw World-Wide-Name(s) (WWN) 5 2.1(2) 1.1 -- 6 2.1(2) 0.602 -- 4-28 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 125
Chapter 4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to [email protected] Mod errorcode: 0x40000002 system time: (1127834823 ticks) Tue Sep 27 15:27:03 2005 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x - Cisco MDS-9124 | Troubleshooting Guide - Page 126
Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to [email protected] Step 5 Note that the time when the module was reinitialized (from system ] Next state: [LCM_ST_LC_FAILURE] 4-30 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 127
coming up, because the XBAR Manager is failing during the insertion of the module. Troubleshooting Reloaded Modules Symptom Module is automatically reloaded. The following system messages may be present if a module reloads: OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-31 - Cisco MDS-9124 | Troubleshooting Guide - Page 128
Switching and Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to [email protected] Error Message MODULE-2-MOD_NOT_ALIVE: Module failure. Recommended Action No action is required. 4-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 129
search for common reload problems. Use the show system time: (1127843486 ticks) Tue Sep 27 17:51:26 2005 error type: FATAL error Number Ports went bad: 1,2,3,4,5,6,7,8 exception information --- exception instance 4 ---- device id: 5 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 130
Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to [email protected] Step 4 device errorcode: 0x40730019 system CLI command to search for common problems. Use the show platform internal Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 131
Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to mdsfeedback-doc@cisco Module is not supported by the current version of Cisco SAN-OS on to search for common problems. Use the show Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-35 - Cisco MDS-9124 | Troubleshooting Guide - Page 132
Services Modules Chapter 4 Troubleshooting Hardware Send documentation comments to [email protected] contents of the module. If the module is supported by the current version of Cisco SAN-OS, the module will be powered-up Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 133
Chapter 4 Troubleshooting Hardware Troubleshooting Switching and Services Modules Send documentation comments to [email protected] Step 6 Step 7 Step 8 switch# reload module the entire switch. switch# reload OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 4-37 - Cisco MDS-9124 | Troubleshooting Guide - Page 134
Troubleshooting Cisco SAN-OS Software System Reboots" section on page 2-13. If you use the module reset-reason CLI command and the output has an "unknown" reset reason, this may indicate a hardware problem to reset or hang. 4-38 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 135
modules supported by the Cisco MDS 9500 Series switches and Cisco MDS 9216A and Cisco MDS 9216i switches, as well as the Generation 2 switches: Table 5-1 Generation 2 Modules and Switches Part Number Modules DS-X9148 DS-9304-18K9 DS-X9112 DS-X9704 DS-X9530-SF2-K9 Switches DS-C9222i-K9 DS-9134-K9 - Cisco MDS-9124 | Troubleshooting Guide - Page 136
Switches (continued) Part Number DS-X9124 DS-9304-K9 Description 24-port 4-Gbps Fibre Channel switch 18-port 1/2/4-Gbps Fibre Channel switch with 4 Gigabit Ethernet ports in fixed slot1 Note Generation 2 Fibre Channel switching modules are not supported on the Cisco MDS 9216 switch; however, they - Cisco MDS-9124 | Troubleshooting Guide - Page 137
cisco.com Table 5-2 Bandwidth and Port Groups for Generation 2 Modules (continued) Module or Switch Description Number of Ports Per Port Group Bandwidth Per Port Group DS-C9124 24-port 4 Gbps 6 12.8 DS-C9222i-K9 only supports 10- Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 138
ports) allocated per Cisco MDS 9000 module. Table 5-6 Port Index Allocation Module 48-port 4-Gbps Fibre Channel switching module 24-port 4-Gbps Fibre Channel switching module Physical Ports 48 24 Port Indexes Allocated 48 24 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-4 OL-9285 - Cisco MDS-9124 | Troubleshooting Guide - Page 139
services module 8 321 4-port Gigabit Ethernet IP Storage services module 4 32-port 2-Gbps Fibre Channel Storage Services Cisco MDS 9500 Series directors. Generation 1 modules also require contiguous port indexes where the system -9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 140
assigned to each slot for Generation 1 modules. 2. 16-port Fibre Channel switching modules can use the upper 16 indexes within a slot (for example, 16-31). Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 141
This is supported on 4-Gbps and 10-Gbps ports. Use these diagnostics to isolate physical layer problems, like contact problems, major failures Cannot Enable a Port, page 5-13 • Cannot Upgrade Supervisor System Image, page 5-13 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 142
) | 8 | 0- 255| - | (None) | 9 | 0- 255| - | (None) | SUP | ----- | 3 | 253-255 | In some cases, the sequence in which switching modules are inserted into the chassis determines if one or more modules is powered up. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 143
-allocation startup Startup module index distribution: Slot | Allowed | Alloted indices info | | range | Total | Index values | 1 | 0- 255| 16 | 64-79 | 2 | 0- 255| 12 | 0-11 | SUP | ----- | 3 | 253-255 | OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 5-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 144
DS-X9148 ok 3 48 1/2/4 Gbps FC Module DS-X9148 ok 4 32 1/2 Gbps FC Module DS-X9032 ok 5 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 active * 6 0 Supervisor/Fabric-1 DS-X9530-SF1-K9 ha-standby 7 16 1/2 Gbps FC Module DS Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 145
of-service mode to free up more resources. b. Right-click a port and select Configure. Lower the port speed. See the "Dynamic Bandwidth Management" section on page 5-3 for the minimum bandwidth requirements for port rate modes and port speeds. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 146
to place in dedicated mode. a. Use the out-of-service command in interface mode to put one or more ports in out- of-service mode to free more resources. b. Use the swtichport requirements for port rate modes and port speeds. 5-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 147
the port in service. Not enough supported by Cisco MDS 9100, 9200 and 9500 Series switches require different system and kickstart images. You can determine which images to use on your switch by the naming conventions shown in Table 5-12. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 148
begins with m9200-s2ek9 Filename begins with m9200-s1k9 Filename begins with m9500-sf1ek9 Filename begins with m9500-sf2ek9 Filename begins with m9500-sf2ek9 5-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 149
Multiprotocol Services (MPS-14/2) module. If you install a module with IP ports in the empty slot on the Cisco MDS 9216i, a separate SAN Extension over IP license is required to enable related features on the IP ports of the additional module. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 150
If you try to use an unlicensed feature, you may see the following system messages: Error Message LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature [chars]. the license to continue using the feature. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-2 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 151
: Step 1 Select Switches > Licenses from the Physical Attributes pane. You see the license information in the Information pane, one line per feature. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 152
Information Using Fabric Manager Web Client Fabric Manager Release 2.1(2) or later supports viewing license use across the fabric from Fabric Manager Web Client. This No 0 Unused 10G_PORT_ACTIVATION_PKG No 0 Unused - Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 153
Permanent.lic: SERVER this_host ANY VENDOR cisco INCREMENT MAINFRAME_PKG cisco 1.0 permanent uncounted \ HOSTID=VDH=FOX0646S017 \ NOTICE="0 \ dummyPak" SIGN=EE9F91EA4B64 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 154
. Serial Number Issues A common problem with licenses stems from not using the correct chassis serial number when ordering your license. To obtain the correct chassis serial number using Fabric Manager, follow these steps: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 155
service representative. Note If you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support at this URL: http://www.cisco -05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 156
. Check your logs for any system messages for a failed license installation both licenses into one license file. • Manually reduce the usage count by one. MDS-1X-JAB-0F1A81" SIGN=F0652E02XXXX INCREMENT SAN_EXTN_OVER_IP_IPS2 cisco 1.0 permanent 1 \ Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 157
MDS switch. To reduce the usage count to one, follow these steps: Step 1 Bring down one of the modules manually has started, you will receive console messages, SNMP traps, system messages, and Call Home messages on a daily basis. Beyond Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 158
of Caveat CSCeg23889, you might still receive Call Home or system messages for an unused FM_SERVER_PKG license. This caveat describes how extraneous messages are sent after a Fabric Manager Server license is checked in. 6-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 159
corrupted. Solutions Reinstall the license. See the "Corrupted Bootflash Recovery" section on page 2-14 to recover from corrupted bootflash:. Reinstall the license. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 6-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 160
Licensing Installation Issues Chapter 6 Troubleshooting Licensing Send documentation comments to [email protected] 6-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 161
cisco.com 7 C H A P T E R Troubleshooting Cisco Fabric Services This chapter describes procedures used to troubleshoot Cisco Fabric Services (CFS) problems in the Cisco MDS • SFM • SDV • syslog • port-security • call home OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 162
Initial Troubleshooting Checklist Chapter 7 Troubleshooting Cisco Fabric Services Send documentation comments to [email protected] As of Cisco MDS SAN-OS Release 3.2(1), the same region on all the affected switches. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-2 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 163
Chapter 7 Troubleshooting Cisco Fabric Services Initial Troubleshooting Checklist Send documentation comments to [email protected] Checklist Verify that there are no pending changes No Physical dpvm Yes Physical OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 164
Initial Troubleshooting Checklist Chapter 7 Troubleshooting Cisco Fabric Services Send documentation comments to [email protected] Step 2 fscm Yes role Scope :Logical [VSAN 3] Domain Switch WWN IP Address Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 165
Chapter 7 Troubleshooting Cisco Fabric Services Merge Failure Troubleshooting Send documentation comments to [email protected] 103 20:00:00:0e:d7:00:3c:9e 10.76. the session lock so that the merge can proceed. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 166
Merge Failure Troubleshooting Chapter 7 Troubleshooting Cisco Fabric Services Send documentation comments to [email protected] Recovering from a Merge Failure with Fabric Manager To :99:c0 LOCK_REQUEST admin 3868 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 167
Chapter 7 Troubleshooting Cisco Fabric Services Lock Failure Troubleshooting Send documentation comments to [email protected] Step 3 Fri Aug 24 04:33:07 2007 20:00 same configuration database and free the CFS lock. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 168
Lock Failure Troubleshooting Chapter 7 Troubleshooting Cisco Fabric Services Send documentation comments to [email protected] Resolving Lock Failure Issues Using the CLI To Switch(config)# ntp abort Switch(config)# Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 169
Troubleshooting Cisco Fabric Services Distribution Status Verification Send documentation comments to [email protected] System State Inconsistent and Locks Being Held An inconsistent system command output follows: OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 170
CFS Regions configuration is not supported for deregistered applications (conditional services) or a physical scope . See "Initial Troubleshooting Checklist" section on page 7-2 for detailed instructions. Verify that the 7-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 171
for the conditional service before it starting it Troubleshooting" section on page 7-5 to identify and resolve the conflicts. Note When an application is moved from one region to another (including the default region), it loses all histories. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 172
CFS Regions Troubleshooting Chapter 7 Troubleshooting Cisco Fabric Services Send documentation comments to [email protected] 7-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 173
administrative states, and operational states, refer to the Cisco MDS 9000 Family Configuration Guide and the Cisco MDS 9000 Fabric Manager Configuration Guide. Initial Troubleshooting Checklist Troubleshooting a SAN problem involves gathering information about the configuration and connectivity of - Cisco MDS-9124 | Troubleshooting Guide - Page 174
out of service. Note Use the show running interface CLI command to view the interface configuration in Cisco SAN-OS Release 3.0(1) or later. The interface configuration as seen in the show running-config CLI command is no longer consolidated. Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 175
merge The zone merge operation failed. failure Isolation due to VSAN mismatch The VSANs at both ends of an ISL are different. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 176
the isolation problem. Limitations and Cisco MDS 9000 Family SAN-OS software. It performs the following functions: • Initialization of FC-MAC ASIC. • Speed negotiation. • Link/loop port initialization and credit recovery. • Statistics collection. Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 177
command to access these FC-MAC show commands. The FLOGI server is a separate application that handles the FLOGI processing for Nx ports. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 178
(pWWN) and Fibre Channel ID (FC ID) of the connected device • For ISLs, the IP address of the connected switch • Speed • Frames transmitted and received Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 179
, pWWN, nWWN, BB credits and class of service for N port connections. • ELP-View pWWN, nWWN, BB credits and supported classes of service for ISLs. • Trunk Config-View and configure trunk BB credits, maximum receive buffer size. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 180
statistics for Class 2 traffic, including busy frame responses and port rejects. • FICON-View FICON error statistics, including pacing, disparity, EOF, OOF, and order sets errors. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 181
either of the ports fails to remain in the online state, then you may have a faulty GBIC, cabling or HBA/subsystem port. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 182
of the FC-MAC Driver and the Port Manager Chapter 8 Troubleshooting Ports Send documentation comments to [email protected] Step 4 Step 5 If both ports are online, that is a member of the network-admin group. 8-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 183
before these problems occur (if possible) and then gather another set of information after these problems occur. Useful Commands at the FC-MAC Level Troubleshooting a port problem involves analysis mac2 port port gbic-info OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 184
of the FC-MAC Driver and the Port Manager Chapter 8 Troubleshooting Ports Send documentation comments to [email protected] Table 8-2 Useful FC-MAC Port Commands (continued) CLI :a0:98:42 50:06:04:82:c3:a0:98:42 8-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 185
in the case of a broken fibre, or if there is no bit synchronization between the switch interface and the directly connected Nx port. This problem may be the result of one or more of the possible causes listed in Table 8-3. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 186
Manager or use the show logging CLI command to check for a Link Failure, Not Connected system message. Right-click on the port in Device Manager and select disable and then enable, of just the VSAN experiencing the problem. 8-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 187
with Port Interfaces Send documentation comments to [email protected] Troubleshooting Port Problems Start the debugging with the command show hardware internal fc speed capabilities and configuration at both ends. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 188
do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem. 8-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 189
this, enter the following command: show flogi database vsan vsan-id Step 3 The system output might look like this: switch# show flogi database vsan 99 INTERFACE VSAN FCID just the VSAN experiencing the problem. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 190
Common Problems with Port Interfaces Chapter 8 Troubleshooting Ports Send documentation comments to [email protected] Step 4 Use the ] Next state: [FLOGI_ST_PERFORM_CONFIG] /* Response from ZoneServer */ 8-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 191
EISL instead of just the VSAN experiencing the problem. Step 6 Use the debug fcns events to FCNS events. The system output may look something like Registered cos 8 for port-id 780200 /* Class of Service */ Feb 17 04:42:54 fcns: vsan 99: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-19 - Cisco MDS-9124 | Troubleshooting Guide - Page 192
Problems with Port Interfaces Chapter 8 Troubleshooting Ports Send documentation comments to mdsfeedback-doc@cisco mode. The system output looks :36 class :3 /* Fibrechannel class of service */ node-ip-addr :0.0.0.0 /* IP Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 193
Chapter 8 Troubleshooting Ports Common Problems with Port Interfaces Send documentation comments to [email protected] Unexpected Link Flapping Occurs Symptom Unexpected link flapping if an ISL is not present. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-21 - Cisco MDS-9124 | Troubleshooting Guide - Page 194
do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem. 8-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 195
Chapter 8 Troubleshooting Ports Common Problems with Port Interfaces Send documentation comments to [email protected] Link Initialization Flow Fibre Channel LR receive substate (LR2) c. LRR receive substate (LR3) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-23 - Cisco MDS-9124 | Troubleshooting Guide - Page 196
bad and if that caused the loss of synchronization. Sometimes this is not necessarily a problem with the physical link, but with the way some devices initialize the link. Use attach . Too many credit loss events occurred. 8-24 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 197
interface all • clear counters interface The counters can identify synchronization problems by displaying a significant disparity between received and transmitted frames. For example, SFP could also cause the issue. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-25 - Cisco MDS-9124 | Troubleshooting Guide - Page 198
states. Table 8-7 lists possible causes and solutions to this problem. Table 8-7 Port Bounces Between the Initializing and Offline States command. See the "Troubleshooting ELP Issues Using the CLI" section on page 8-27. 8-26 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 199
8 Troubleshooting Ports Common Problems with Port Interfaces Send documentation comments to [email protected] Troubleshooting ELP Issues Using the CLI To troubleshoot ELP beacon(off), snmp trap(on), tem(false) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-27 - Cisco MDS-9124 | Troubleshooting Guide - Page 200
lists possible causes and solutions to this problem. Table 8-8 E Port Remains Isolated Troubleshooting E port Isolation using Fabric Manager" section on page 8-29 or the "Troubleshooting E port Isolation using Fabric Manager" section on page 8-29. 8-28 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 201
To work around this, you can manually change the content of the zone database of just the VSAN experiencing the isolation problem. Using the Zone Merge Analysis tool Cisco MDS 9000 Fabric Manager Configuration Guide for more information. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 202
are overwritten, you cannot use the import option. To work around this, you can manually change the content of the zone database on either of the switches, and then issue just the VSAN experiencing the isolation problem. 8-30 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 203
the CLI, follow these steps: Step 1 Use the show interface command to verify that the switch detected a problem and disabled the port. Check cables, SFPs, and optics. mds# show interface fc1/14 fc1/14 is down (errDisabled) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 8-31 - Cisco MDS-9124 | Troubleshooting Guide - Page 204
added to port-channel 7 Troubleshooting Fx Port Failure Fx port problems can be caused by troubleshooting include: • Why is no storage visible on my newly installed server? • Why is previously assigned storage not visible to my server after reboot? 8-32 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 205
Chapter 8 Troubleshooting Ports Common Problems with Port Interfaces Send documentation comments to [email protected] Troubleshoot physical ports Troubleshoot HBA Check VSAN membership Check zoning config and LUN masking 144870 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 206
Common Problems with Port Interfaces Chapter 8 Troubleshooting Ports Send documentation comments to [email protected] 8-34 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 207
log in as an N-port to utilize this feature (loop-attached devices are not supported). All links from the edge switches (in NPV mode) to the NPV core switches of NPV, refer to the Cisco MDS 9000 Family CLI Configuration Guide. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 208
cisco.com Initial Troubleshooting Checklist Troubleshooting an NPV problem • NPV core switches must support NPIV. • You cannot manually assign the server interfaces to MDS switch running SAN-OS 3.2(1) or later. The switch does not need to be NPV enabled. Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 209
N-Port Virtualization Initial Troubleshooting Checklist Send documentation comments to [email protected] To display the status and VSAN membership of the different servers ff:00:06:2b:10:c7:b3 (LSI) scsi-fcp:init OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 210
Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to [email protected] Total number of npv-attached entries = 2 If you need to contact support, enter the show tech-support NPV command and save the output so that support can use it to - Cisco MDS-9124 | Troubleshooting Guide - Page 211
Chapter 9 Troubleshooting N-Port Virtualization Common Problems with NPV Send documentation comments to [email protected] Step 2 Enter the no shutdown command on the same to have different VSAN on both ends. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 212
Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to [email protected] Correct the VSAN configuration. The VSAN to ensure that the port is properly connected. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 213
Chapter 9 Troubleshooting N-Port Virtualization Common Problems with NPV Send documentation comments to [email protected] Step 2 Check to ensure that the port mode on the NPV must be up for server interface to be up. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 214
Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to [email protected] Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 215
Send documentation comments to [email protected] 10 C H A P T E R Troubleshooting PortChannels and Trunking PortChannels refer to the aggregation of multiple channel group. This number ranges from of 1 to 256. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 216
: • Choose ISLs > PortChannel to access the PortChannel configuration. • Choose Switches > Interfaces > FC Logical and select the Trunk Config tab to access the trunking configuration. 10-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 217
if you want to manually configure PortChannels. In Device Manager, select Interfaces > FC ALL..., select the Other tab, uncheck the AutoChannelCreate check box, and click Apply. Use the no channel-group auto CLI command. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 218
page 10-4. Or, use the no shutdown CLI command to enable the PortChannel manually or use the channel-mode active CLI command in the interface submode for the 10-5 • VSAN Traffic Does Not Traverse Trunk, page 10-5 10-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 219
, select the Trunk Config tab, and set the Allowed VSANs field. Click Apply Changes. Use the switchport trunk allowed vsan CLI command. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 10-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 220
Trunking Issues Chapter 10 Troubleshooting PortChannels and Trunking Send documentation comments to [email protected] 10-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 221
. • Enable more scalable and secure fabrics. Initial Troubleshooting Checklist Most VSAN problems can be avoided by following the best practices for Cisco MDS 9000 Fabric Manager Configuration Guide for more information about this tool. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 222
member ports in a PortChannel becoming suspended or error-disabled. Troubleshooting a SAN problem involves gathering information about the configuration and connectivity of individual vsan membership interface fc slot/port 11-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 223
Port Is Isolated in a VSAN, page 11-5 • Troubleshooting Interop Mode Issues, page 11-9 Host Cannot Communicate with Storage Communication problems between a host and storage devices can be caused by on page 14-4. zone. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 224
VSAN Issues Chapter 11 Troubleshooting VSANs, Domains, and FSPF Send documentation comments to [email protected] Verifying VSAN Membership Using Fabric Manager To verify VSAN membership mode is TE Speed is 2 Gbps 11-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 225
Issues Using the CLI" section on page 11-9. Port parameters misconfigured. See the "Common Problems with Port Interfaces" section on page 8-13. Zoning mismatch. See Chapter 14, "Troubleshooting Zones and Zone Sets." OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 226
VSAN Issues Chapter 11 Troubleshooting VSANs, Domains, and FSPF Send documentation comments to [email protected] Resolving an Isolated E Port Using Fabric Manager To resolve VSAN move the ports into the same VSAN. 11-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 227
and check the FailureCause column on the TE port to verify that you have a trunk problem. Choose Switches > Interfaces > FC Physical and select the Trunk Failures tab to determine the reason for port has one VSAN isolated. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 228
Not Activating" section on page 11-14 for domain misconfiguration problems. Use the vsan vsan-id interface command to correct the VSAN misconfiguration problems. Repeat this procedure for all isolated VSANs on this TE port. 11-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 229
Attributes to verify that the fabric timers are inconsistent across the VSANs. Choose Switches > FC Services > Timers and Policies. You see the fabric timers in the Information pane. Click Change is connected or moved. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 230
DPVM, refer to one of the following guides: • Cisco MDS 9000 Family Fabric Manager Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide This section contains the following topics: • Troubleshooting DPVM Using Fabric Manager, page 11-10 • Troubleshooting DPVM Using the CLI, page 11-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 231
Status field to enable in Fabric Manager and then click Apply Changes or use the dpvm enable CLI command to enable DPVM. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 232
with these autolearned entries. When you add a new device, it is a best practice to manually add that device to the DPVM configuration database. If you turn on autolearning for a new activate and autolearn to take effect. 11-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 233
that the device you want to autolearn is connected to an F port. DPVM does not support FL, TE, FCIP, or PortChannels. VSAN Membership not Added to Database Symptom The VSAN membership of commit any pending changes. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 234
Changes. Or use the dpvm activate and dpvm commit CLI commands to create the DPVM active database. Then copy the active database again. 11-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 235
dpvm CLI command to verify the DPVM configuration in both fabrics. Manually reconcile any differences before attempting to merge the fabrics. Use the show cfs merge status name dpvm CLI command to verify the merge status. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 236
VSAN on that port, ensuring that ports continue to be in the same VSANs and thus maintaining service continuity. To disable DPVM, follow these steps: Note No devices should login or logout during this on the switch. 11-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 237
is used by every switch), the principal switch is determined by the WWN of the existing principal switches of the two fabrics. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 238
ID overlap, you can either change the overlapping static domain ID by manually configuring a new static domain ID for the isolated switch, or page 11-20. You may see the following system message in the message log when a domain Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 239
, 16777216 discards Received 28 OLS, 19 LRR, 16 NOS, 48 loop inits Transmitted 62 OLS, 22 LRR, 25 NOS, 30 loop inits OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-19 - Cisco MDS-9124 | Troubleshooting Guide - Page 240
a new principal switch selection and cause new domain IDs to be assigned to the different switches. Caution A disruptive reconfiguration might affect data traffic. 11-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 241
(100) (preferred) Principal switch run time information: Running priority: 2 Interface fc2/1 fc2/2 fc2/7 Role Downstream Downstream Upstream RCF-reject -----------Enabled Disabled Disabled OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-21 - Cisco MDS-9124 | Troubleshooting Guide - Page 242
switches in the VSAN to determine what domain IDs are missing. Use the fcdomain allowed CLI command to add any missing domain IDs. 11-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 243
Allowed domain ID list incorrect after a VSAN merge Possible Cause Solution Domain ID lists not manually updated before the merge. Add all domain IDs in the VSAN to the domain ID list fcdomain restart CLI command. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-23 - Cisco MDS-9124 | Troubleshooting Guide - Page 244
FSPF Issues Chapter 11 Troubleshooting VSANs, Domains, and FSPF Send documentation comments to [email protected] FSPF Issues The implementation of VSANs dictates that each configured VSAN support a separate set of fabric services. One such service is the FSPF routing protocol, which can - Cisco MDS-9124 | Troubleshooting Guide - Page 245
1 1000 - Cisco MDS-9124 | Troubleshooting Guide - Page 246
is 1000 - Cisco MDS-9124 | Troubleshooting Guide - Page 247
Up to sixteen paths can be used by FSPF with a Cisco MDS 9000 Family switch. With the implementation of VSANs used with Cisco MDS 9000 Family switches, a separate instance of FSPF runs within in the statement or command OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-27 - Cisco MDS-9124 | Troubleshooting Guide - Page 248
FSPF Issues Chapter 11 Troubleshooting VSANs, Domains, and FSPF Send documentation comments to [email protected] Loss of Two-Way Communication If FSPF is misconfigured, change the hello interval and click Apply. 11-28 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 249
a problem. Repeat Step 4 to determine the value of the Hello timer on the adjacent switch. switch2# show fspf v 1 interface fc2/16 FSPF interface fc2/16 in VSAN 1 FSPF routing administrative state is active Interface cost is 500 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 250
is not set to the default, so you should check the neighbor configuration to make sure it matches. 2. FSPF is not in FULL state, indicating a problem. 11-30 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 251
is set to the default (5 seconds). 2. FSPF is not in FULL state, indicating a problem. Use the interface command and then the fspf retransmit-interval command in interface mode to change all command to turn off debugging. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-31 - Cisco MDS-9124 | Troubleshooting Guide - Page 252
should check the neighbor configuration. 2. FSPF is not in full state, which indicates a problem. Use the interface comma nd and then the fspf dead-interval command in interface mode of LSRs = 2, Total Checksum = 0x000133de 11-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 253
FSPF configuration mode and use the region command to change the region. The region must match on all switches in the VSAN. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 11-33 - Cisco MDS-9124 | Troubleshooting Guide - Page 254
FSPF Issues Chapter 11 Troubleshooting VSANs, Domains, and FSPF Send documentation comments to [email protected] 11-34 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 255
and uses CFS (Cisco Fabric Services) distribution to synchronize the databases. Initial Troubleshooting Checklist Begin troubleshooting SDV issues by checking the following issues: Checklist Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Enable SDV on - Cisco MDS-9124 | Troubleshooting Guide - Page 256
internal acl-cache dump show sdv zone active debug sdv all-sdv show tech-support sdv Description Displays the SDV virtual device configuration that has been committed. Displays the entries because they have the same name. 12-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 257
with the device aliases currently used in the fabric. SDV Issues This section describes potential problems associated with SDV, and includes the following topics: • SDV Commit Fails, page 12 to take when an SDV commit fails. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 12-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 258
reserved. The failing switches produce a syslog identifying the reason for the failure. View the syslog and identify the reason for the failure; rectify the problem and reissue the commit command. CFS distribution failure. 12-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 259
the supported topologies for SDV to work. Between the host, the SDV, and the real device, there must be at least one of the following switches in the path: • 95xx • 9216 • 9216A • 9216i • 9120 • 9140 SDV must also be enabled. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 260
mapping maintained in the zone server. Devices should be zoned so that they cannot communicate to both the real device and its virtual component. 12-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 261
Troubleshooting IVR involves checking the configuration of domain IDs, VSANs, border switches, and zone sets. Configuration problems with IVR can prevent devices from communicating properly. Prior to Cisco MDS , page 13-2 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 262
is not supported on the Cisco MDS 9124 Fabric Switch, the Cisco Fabric Switch for HP c-Class BladeSystem, and the Cisco Fabric Switch Cisco MDS SAN-OS Release 3.x" for complete limitations to the IVR configuration based on the Cisco SAN-OS release.) 13-2 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 263
32 IVR zone sets per physical fabric. Initial Troubleshooting Checklist Begin troubleshooting IVR issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that IVR is enabled on all border - Cisco MDS-9124 | Troubleshooting Guide - Page 264
list or manually configure static, service-group Shows the difference between the IVR pending and configured databases. show ivr tech-support Shows information that is used by your customer support representative to troubleshoot Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 265
Table 13-2 IVR Enhancements by Cisco SAN-OS Release Cisco SAN-OS Release Release 3.3(1) Release 2.1(2) IVR Enhancement Support for read-only LUN attribute in IVR zone configuration. Persistent FC IDs and domains for IVR OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 266
• IVR QoS zoning • Service group IVR with CFS support Virtual domains added to remote domain lists IVR introduced IVR Issues This section describes the problems associated with IVR. This or chassis you have enabled IVR on. 13-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 267
2, "Troubleshooting Installs, Upgrades, and Reboots." Choose Fabricxx > All VSANs > IVR, set the Global drop-down menu to enable, and click Apply Changes in Fabric Manager. Or use the ivr distribute CLI command before enabling IVR. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 268
Fails If zone set activation fails, you may see the following system messages: Error Message IVR-2-IVZS_ACTIVATION_FAILED_RETRYING: Inter-VSAN zoneset activation failed in is required. Introduced Cisco MDS SAN-OS Release 1.3(1). 13-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL- - Cisco MDS-9124 | Troubleshooting Guide - Page 269
Troubleshooting IVR IVR Issues Send documentation comments to mdsfeedback-doc@cisco required. Introduced Cisco MDS SAN-OS . Introduced Cisco MDS SAN-OS action is required. Introduced Cisco MDS SAN-OS Release 2.0(1b set has been activated. See the "Troubleshooting Zone Set Activation" section on page - Cisco MDS-9124 | Troubleshooting Guide - Page 270
. Or use the ivr vsan topology auto CLI command to automatically reconfigure the IVR topology, or use the ivr vsan topology database CLI command to manually reconfigure the IVR topology. 13-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 271
the Service Group tab in Fabric Manager. Or use the ivr service-group CLI command. Use the ivr service-group activate CLI command to activate this change. If CFS is enabled, use the ivr commit CLI command to commit this change. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 272
use the pwwn CLI command to map the pWWN to an appropriate FC ID that matches the virtual domain ID. Refer to the Cisco MDS 9000 Family configuration guides for the related procedure to configure Persistent FC IDs for IVR. 13-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 273
distribute and update the IVR VSAN topology on all switches. In rare cases, you may encounter problems where CFS locks IVR so that you cannot modify the configuration. Symptom Locked IVR CFS session. free the session lock. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 274
Action If you purchased Cisco support through a Cisco reseller, contact the reseller directly. If you purchased support directly from Cisco Systems, contact Cisco Technical Support. Introduced Cisco MDS SAN-OS Release 2.0(1b). 13-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285 - Cisco MDS-9124 | Troubleshooting Guide - Page 275
reconfigure the IVR topology, or the ivr vsan topology database CLI command to manually reconfigure the IVR topology. Reconfigure your fabric before merging to reduce the number . Please Retry the Request Later, page 13-17 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 276
Apply Changes. This triggers a disruptive restart to make the running domain ID match the configured domain ID. Use IVR NAT. This may require upgrading to Cisco MDS SAN-OS Release 2.1(1a) or later. 13-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 277
Is Changing. Please Retry the Request Later This error may occur if there are different versions of Cisco SAN-OS on the IVR-enabled switches. You should upgrade all IVR-enabled switches to the same version of Cisco SAN-OS. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 13-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 278
Troubleshooting the IVR Wizard Chapter 13 Troubleshooting IVR Send documentation comments to [email protected] 13-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 279
Zone Sets This chapter describes how to identify and resolve problems that might occur while implementing zones and zone sets on switches in the Cisco MDS 9000 Family. It includes the following sections: • Overview, page 14-1 • Troubleshooting Checklist, page 14-1 • Zone and Zone Set Issues, page - Cisco MDS-9124 | Troubleshooting Guide - Page 280
com For zone configuration problems, use the following helpful tools: • Cisco Fabric Analyzer. (See the "Cisco Fabric Analyzer" section on page B-25.) • Cisco Fabric Manager and CLI system messages. (See the System Messages, page 1-10.) • Log messages (See the "Troubleshooting with Logs" section on - Cisco MDS-9124 | Troubleshooting Guide - Page 281
relevant zoning information that may be requested by your customer support representative when troubleshooting zoning issues. Note To issue commands with the internal of IPS zones: 0 Formattted size: 38 bytes / 2048 Kb OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 282
aliases: 0 Num attribute groups: 0 Formattted size: 20 bytes / 2048 Kb See the Cisco MDS 9000 Family Command Reference for the description of the information displayed in the command output. The the Same VSAN, page 14-17 14-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 283
in the active zone set, see the "Resolving Zone is Not in Active Zone Set Using Fabric Manager" section on page 14-6. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 284
-4. Configure zoning, if necessary, by using the show zone status vsan-id command to determine if the default zone policy is set to deny. 14-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 285
show zone command to verify that host and storage are now in the same zone. switchA# show zone zone name NewZoneName vsan 2 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 286
enforce zoning, and is called the active zone set. A zone that is part of an active zone set is called an active zone. Two main problems can occur with activating a zone set: • No zone set is active. 14-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 287
experience timeout errors in Cisco MDS SAN-OS Release 1.3(4a) and earlier. When a zone set activation fails, you may see the following system messages: Error Message to determine the source of zone set activation problem. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 288
Zones and Zone Sets Send documentation comments to [email protected] Troubleshooting Zone Activation Using the CLI To verify the active zone set and active vsan 2 switch(config-zoneset)# member NewZoneAdded 14-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 289
determine the source of the zone set activation problem. Troubleshooting Full Zone Database Synchronization Across Switches All switches in the Cisco MDS 9000 Family distribute active zone sets when distribute flag is on. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 290
VSAN. If the default zone policy varies, then you may experience zoning problems. If all switches in the VSAN have Cisco SAN-OS Release 2.0(1b) or later, you can use enhanced zoning. that you save the running configuration. 14-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 291
verify that all switches are capable of working in the enhanced mode. All switches must have Cisco MDS SAN-OS Release 2.0(1b) or later. If one or more switches are not capable of Too many zone sets • Too many aliases OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 292
by the interface. Databases are not merged. Recommended Action Compare full zoning database with the adjacent switch, correct the difference and flap the link. Introduced Cisco MDS SAN-OS Release 1.2(2a). 14-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 293
current database to the neighboring switch. • Manually resolve the conflict by editing the full Problems in VSAN Id field. Click Analyze to analyze the zone merge. Click Clear to clear the analysis data from the Zone Merge Analysis dialog box. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 294
with that of the remote switch. The export option overwrites the remote switch's active zone set with the local switch's active zone set. 14-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 295
affect all the VSANs crossing the EISL instead of just the VSAN experiencing the isolation problem. Mismatched Active Zone Sets Within the Same VSAN When merging switch fabrics, you must missing zone within the zone set. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 296
Chapter 14 Troubleshooting Zones and Zone Sets Send documentation comments to [email protected] Step 3 Do one of the following to resolve the isolation problem: • Change 0 unknown class 0 too long, 0 too short 14-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 297
VSAN is isolated because of o a zone merge failure. Do one of the following to resolve the isolation problem: • Change the membership of one of the zones to match the other zone of the same name. 99 is up, FCID is 0x780102 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-19 - Cisco MDS-9124 | Troubleshooting Guide - Page 298
enable the connection to the zone to be merged. You may see the following system messages: Nov 19 10:28:11 switch4 %LOG_PORT_CHANNEL-5-FOP_CHAN GED: port-channel cause the MDS 9000 switch to lose connectivity with the network. 14-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285- - Cisco MDS-9124 | Troubleshooting Guide - Page 299
prevent others from making zoning changes. The user must issue a commit to make the changes active and release the fabric wide lock. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-21 - Cisco MDS-9124 | Troubleshooting Guide - Page 300
Enhanced Zoning Issues Chapter 14 Troubleshooting Zones and Zone Sets Send documentation comments to [email protected] Problems can occur when the lock is acquired, but stating that the lock is currently busy. 14-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 301
valid configuration change is in progress before you clear a lock. Step 4 If problems persist, use the clear zone lock command to remove the lock from the switch. This should only be done on the switch that holds the lock. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 14-23 - Cisco MDS-9124 | Troubleshooting Guide - Page 302
Enhanced Zoning Issues Chapter 14 Troubleshooting Zones and Zone Sets Send documentation comments to [email protected] 14-24 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 303
can be used in all configuration commands as required. All switches in the Cisco MDS 9000 Family support Distributed Device Alias Services (device aliases). Initial Troubleshooting Checklist Device alias problems may result in merge, commit, or validation failures, and resolution requires gathering - Cisco MDS-9124 | Troubleshooting Guide - Page 304
comments to [email protected] Note See the "System Messages" section on page 1-10 for more information about using system (syslog) messages for troubleshooting purposes. This section also identifies some of the system messages that are issued when device alias problems are encountered, and - Cisco MDS-9124 | Troubleshooting Guide - Page 305
configuration information for the device alias service, including whether fabric distribution is supported on switches running SAN-OS Release 3.1(1) or later. • Interop mode VSANs do not accept enhanced mode (native alias-based) configurations. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 306
15 Troubleshooting Distributed Device Alias Services Send documentation comments to [email protected] the supported limit, then the merge will fail. Merge Failure Issues The most common device alias problems occur Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 307
Chapter 15 Troubleshooting Distributed Device Alias Services Merge Failure Issues Send documentation comments to [email protected] Resolving Refer to the Cisco MDS 9000 Family CLI Configuration Guide for details. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 308
Failure Issues Chapter 15 Troubleshooting Distributed Device Alias Services Send documentation comments to [email protected] Resolving Mapping a 3.1(1) or later to support device alias enhanced mode operation. 15-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 309
switches in the problem fabric to SAN-OS Release 3.1(x) or later, and then attempt the merge again. Otherwise, if you must merge the fabrics, you will have to reduce the number of entries in the combined device alias database. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15 - Cisco MDS-9124 | Troubleshooting Guide - Page 310
Troubleshooting Distributed Device Alias Services Send documentation comments to mdsfeedback-doc@cisco shows the reason for the failure, along with the system application (SAP) number. switch-1# show device-alias internal -8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 311
Troubleshooting Distributed Device Alias Services Merge Failure Issues Send documentation comments to mdsfeedback-doc@cisco. was the IVR process. switch-2# show system internal mts sup sap 110 description IVR-SAP Step 6 Analyze Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 312
Merge Failure Issues Chapter 15 Troubleshooting Distributed Device Alias Services Send documentation comments to [email protected] Because the application is busy, it cannot SAPS: 0 MTS Buffer Array Details: 15-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 313
Chapter 15 Troubleshooting Distributed Device Alias Services Validation and Commit Failure Issues Send documentation comments to [email protected] Buffers: problems that may cause validation and commit failures. See Table 15-3. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, - Cisco MDS-9124 | Troubleshooting Guide - Page 314
Troubleshooting Distributed Device Alias Services Send documentation comments to mdsfeedback-doc@cisco Size Issues" section on page 15-14. Enhanced mode is not supported on some switches in the mixed fabric. Upgrade the SAN-OS Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 315
Troubleshooting Distributed Device Alias Services Validation and Commit Failure Issues Send documentation comments to mdsfeedback-doc@cisco in IVR. IVR detects the configuration problem and rejects the device alias database validation Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 15-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 316
Chapter 15 Troubleshooting Distributed Device Alias Services Send documentation comments to [email protected] Check ivr support either the issued command or maximum device-alias limits. Please fix those switches and retry the command. 15-14 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 317
15 Troubleshooting Distributed Device Alias Services Validation and Commit Failure Issues Send documentation comments to [email protected] which cannot support enhanced mode. Please fix those switches and retry the command. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 318
Validation and Commit Failure Issues Chapter 15 Troubleshooting Distributed Device Alias Services Send documentation comments to [email protected] 15-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 319
[email protected] 16 C H A P T E R Troubleshooting FICON Fibre Connection (FICON) interface capabilities enhance the Cisco MDS 9000 Family by supporting both open systems and mainframe storage network environments. Inclusion of Control Unit Port (CUP) support further enhances the MDS offering - Cisco MDS-9124 | Troubleshooting Guide - Page 320
Overview Chapter 16 Troubleshooting FICON Send documentation comments to [email protected] Note The FICON feature is not supported on Cisco MDS 9120, 9124 or 9140 switches, the 32-port Fibre Channel switching module, Cisco Fabric Switch for HP c-Class BladeSystem or Cisco Fabric Switch - Cisco MDS-9124 | Troubleshooting Guide - Page 321
-port, or 24-port module and the rest remain unused. Extra 16 ports on 48-port modules are not allocated port numbers. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 322
numbers. Note FICON port numbers are not changed for ports that are active. You must first disable the interfaces using the shutdown CLI command. 16-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 323
written by IBM hosts using the in-band CUP protocol. Additionally, you can use the Cisco MDS CLI or Fabric Manager applications to modify these FICON configuration files. When you enable the FICON port numbers to the module. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 324
IBM. CUP is supported by switches and directors in the Cisco MDS 9000 Family. The CUP function allows the mainframe to manage the Cisco MDS switches. Host communication -enabled VSAN. • Set the VSAN interop value to default. 16-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 325
can only be in the default interop mode. Initial Troubleshooting Checklist Begin troubleshooting FICON issues by checking the following issues: Checklist Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that you enabled in-order delivery for the - Cisco MDS-9124 | Troubleshooting Guide - Page 326
port swapping feature: • Port swapping is not supported for logical ports (PortChannels, FCIP links). • Port swapping is not supported between physical ports that are part of a ), the port swapping operation is rejected. 16-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 327
Chapter 16 Troubleshooting FICON FICON Issues Send documentation comments to [email protected] Swapping Ports To swap ports, except the port numbers, using the -12 • FICON Tape Acceleration Not Working, page 16-12 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 328
FICON Issues Chapter 16 Troubleshooting FICON Send documentation comments to [email protected] Cannot Enable FICON Symptom Cannot enable FICON. Table 16-2 FICON. submode to enable SNMP control for FICON. 16-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 329
> Port Numbers in Device Manager. Alternatively, use the ficon slot slot number assign port-numbers CLI command to change the port number. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 16-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 330
FICON Issues Chapter 16 Troubleshooting FICON Send documentation comments to [email protected] Cannot Configure FCIP or PortChannel for FICON Symptom tape-accelerator vsan CLI command on each FCIP interface. 16-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 331
users managing a switch. All Cisco MDS 9000 Family switches use the Remote Access Dial-In User Service (RADIUS) and Terminal Access Controller Access Control System Plus (TACACS+) protocols to passwords configured. Check off OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 332
• User Cannot Access Certain Features, page 17-11 Switch Does Not Communicate with AAA Server Multiple misconfigurations can result in an AAA server that the Cisco SAN-OS switch does not communicate with. 17-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 333
Configuration Using Fabric Manager" section on page 17-7 or the "Verifying TACACS+ Server Monitor Configuration Using the CLI" section on page 17-7. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 334
servers before the switch declares a timeout failure. Use the radius commit command to commit any changes and distribute to all switches in the fabric. 17-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 335
the switch declares a timeout failure. Use the tacacs commit command to commit any changes and distribute to all switches in the fabric. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 336
the switch waits before retesting a dead server. Use the radius commit command to commit any changes and distribute to all switches in the fabric. 17-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 337
waits before retesting a dead server. Use the tacacs commit command to commit any changes and distribute to all switches in the fabric. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 338
TACACS+ Server Groups Using Fabric Manager" section on page 17-9 or the "Verifying TACACS+ Server Groups Using the CLI" section on page 17-10. 17-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 339
the Deadtime field to configure the time that the switch waits before retesting a dead server. and click Apply to save these changes. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 340
the role using: Cisco-AVPair = shell:roles="rolename1 rolename2". For TACACS+, configure the attribute and value pair on the server for the role using: roles="rolename1 rolename2". Verify that all roles are defined on the switch. 17-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL - Cisco MDS-9124 | Troubleshooting Guide - Page 341
authentication attempts. Choose Reports and Activity to view the resulting logs. On the Cisco SAN-OS switch, use the show radius-server command to verify that the RADIUS server timeout value is set to 5 seconds or greater. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 17-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 342
comments to [email protected] Refer to the User guide for Cisco Secure ACS at the following website for more information: http://cisco.com/en/US/products/sw/secursw/ps2086/products_user_guide_list.html 17-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 343
or Device Manager) and vice versa. User Accounts Every Cisco MDS 9000 Family switch user has the account information stored by the system. You can add up to 256 users to a . The following examples show strong passwords: OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 344
mailnull, rpc, rpcuser, xfs, gdm, mtsuser, ftpuser, man, and sys. Caution Cisco MDS SAN-OS does not support all numeric user names, whether created with TACACS+ or RADIUS, or created locally. to the required commands. 18-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 345
rule next, you would be allowing the user to perform all configuration commands because the second rule globally overrode the first rule. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 346
documentation comments to [email protected] Initial Troubleshooting Checklist Begin troubleshooting user and role issues by checking the following issues: Checklist Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that the passwords for - Cisco MDS-9124 | Troubleshooting Guide - Page 347
the user if necessary. Verifying User Login with System Messages Using Device Manager To configure the switch logging to capture system messages when a user attempts to log into a debug information for these facilities. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 348
log file. switch(config)# logging logfile TestFile 7 This sets the switch to save system messages at the debug level or above in the TestFile log file. At this point, all future login attempts are tracked in the log file. 18-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 349
If you have logged in as a network-admin using MDS authentication, Device Manager and Fabric Manager automatically provide the appropriate encryption for this task, even if you did not specify a specific privacy password. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 350
User and Role Issues Chapter 18 Troubleshooting Users and Roles Send documentation comments to [email protected] User Cannot Access Certain Features Symptom User access and click Apply to save these changes. 18-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 351
the rules assigned to a role. switch# role name sangroup switch(config-role)# no rule 4 switch(config-role)# rule 4 deny exec feature fcping OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 352
User and Role Issues Chapter 18 Troubleshooting Users and Roles Send documentation comments to [email protected] User Has Too Much Access Symptom User has too much to configure in the Scope VSAN Id List field. 18-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 353
Using Fabric Manager" section on page 18-10 or the "Verifying VSAN-Restricted Roles Using the CLI" section on page 18-11. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 18-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 354
timeout value is set to 5 seconds or greater. Refer to the User guide for Cisco Secure ACS at the following website for more information: http://cisco.com/en/US/products/sw/secursw/ps2086/products_user_guide_list.html 18-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 355
Protocol (DHCHAP) is an FC-SP protocol that provides authentication between Cisco MDS 9000 Family switches and other devices. You can configure FC-SP to authenticate locally or to use a remote AAA server for authentication. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 356
with any domain ID. Initial Troubleshooting Checklist Begin troubleshooting FC-SP issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that your installed HBAs support FC-SP. Verify that - Cisco MDS-9124 | Troubleshooting Guide - Page 357
history vsan Us e the following CLI commands to troubleshoot fabric binding issues: • show fabric-binding status • show fabric-binding database vsan • show fabric-binding database active vsan • show fabric-binding violations OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 358
"Verifying Local FC-SP Database Using the CLI" section on page 19-6. Host or switch does not support FC-SP. Upgrade host or switch or use the auto-active or auto-passive DHCHAP mode. Choose mode to set the DHCHAP mode. 19-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 359
Choose Switches > FC Services > WWN Manager to find the sWWN for the switch. Choose Switches > Interfaces > FC Logical and select the FLOGI tab to find the pWWN for the host that you want to add to the FC-SP local database. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 360
to add a host or switch to the local database. switch(config)# fcsp dhchap devicename 20:00:00:33:8b:00:00:00 password rtp9509 19-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 361
Cisco ACS Symptom Authentication fails when using Cisco ACS. Possible Cause sWWN does not match ACS entry. Solution Verify the sWWN and ACS entry. Choose Switches > FC Services reactivation if the problem was fixed. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 362
or the "Configuring Port Security with Autolearn Using the CLI" section on page 19-15. Manually add the device to the configured port security database. See the "Verifying the Active Port the CLI" section on page 19-11. 19-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 363
-security activate command to copy the configure database to the active database and reactivate port security. switch(config)# port-security activate vsan 1 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 364
, if CFS is enabled, and select commit from the ConfigAction drop-down menu to distribute these changes to all switches in the fabric. 19-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 365
port back online. Optionally, remove the device from the switch and use the no shutdown command to bring the port back online. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 366
the "Disabling Autolearn Using Fabric Manager" section on page 19-13 or the "Disabling Autolearn Using the CLI" section on page 19-13. 19-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 367
"Disabling Autolearn Using Fabric Manager" section on page 19-13 or the "Disabling Autolearn Using the CLI" section on page 19-13. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 368
startup configuration, using the fabric option. This saves the port security configure database to the startup configuration on all switches in the fabric. 19-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 369
binding configuration issue, you do not have to disable the interface and reenable it. The port comes up automatically after a fabric binding reactivation if the problem was fixed. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 370
port back online. Click Apply Changes. Note You may need to set the interface down and then up to bring it back online. 19-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 371
fabric binding. switch(config)# fabric-binding activate vsan 3 d. Use the no shutdown command in interface mode to bring the port back online. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 372
fabric-binding activate command to copy the configure database to the active database and reactivate fabric binding. switch(config)# fabric-binding activate vsan 1 19-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 373
configuration, using the fabric option. This saves the port security configure database to the startup configuration on all switches in the fabric. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 19-19 - Cisco MDS-9124 | Troubleshooting Guide - Page 374
startup configuration, using the fabric option. This saves the fabric binding configure database to the startup configuration on all switches in the fabric. 19-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 375
cisco.com 20 C H A P T E R Troubleshooting IP Storage Services This chapter describes how to identify and resolve IP storage services problems that might occur in the Cisco MDS allows you to use FCIP and iSCSI features. It supports the full range of features available on other switching modules - Cisco MDS-9124 | Troubleshooting Guide - Page 376
Overview Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] 94218 Figure 20-1 Connecting MDS 9000 Family Switches Over IP MDS1 FC FC1/14 GE 2/8 10.10.10.2/24 IP network MDS2 GE 2/8 10.10.11.2/24 FC1/1 FC HBA - Cisco MDS-9124 | Troubleshooting Guide - Page 377
Manager, which supports iSLB with CFS distribution. Initial Troubleshooting Checklist Begin troubleshooting IP storage services issues by checking the following issues: Checklist Check off Verify licensing requirements. See the Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify - Cisco MDS-9124 | Troubleshooting Guide - Page 378
logfile that captures system messages from all modules. Use the following commands as directed by your customer support representative to further troubleshoot iSLB issues: Note commands to gather more information for iSLB: 20-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 379
Chapter 20 Troubleshooting IP Storage Services IP Issues Send documentation comments to [email protected] • debug ips error • debug ips islb [config | config-detail | flow | flow- : icmp_seq=0 ttl=128 time=0.3 ms OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 380
IP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Step 2 Step 3 Step 4 Step 7805631 bytes, 0 underruns 0 output errors, 0 collisions, 0 fifo 0 carrier errors 20-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 381
Chapter 20 Troubleshooting IP Storage Services IP Issues Send documentation comments to [email protected] Verification of Switch Connectivity Use the procedures in this 0xda(218), the concatenated ID is 0xfffcda. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 382
IP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Step 3 Use the fcping command to , gigabitethernet8/8 S 11.2.56.0/22 via 11.3.56.1, gigabitethernet8/8 20-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 383
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] Cannot Assign IP Address to an Interface You may encounter a problem when write or read operations. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 384
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] One-to-One FCIP Tunnel Creation and Monitoring This a profile to use. MDS1(config-if)# use-profile 28 20-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 385
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] The FCIP interface will use the local FCIP profile. on a separate terminal. MDS2(config)# fcip profile 28 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 386
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Mar 10 21:41:04 ips: Dequeued mts msg to entity 28 port: 3225 ip: 10.10.11.2 20-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 387
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] MDS2(config-if)# peer-info ipaddr msg MTS_OPC_PM_LOGICAL_PORT_STATE_CHANGE_RANGE(msg_id 47540) dropped OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 388
Send LINK UP to SUP Mar 13 19:18:20 port8: 2715:FCIP28: *** Received eisl frame in E mode Mar 13 19:18:20 port8: 2716:FCIP28: SUP-> Set trunk mode: 2 Mar 13 19:18:20 port8: 2717:FCIP28: Change the operational mode to TRUNK 20-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 389
frame for verification of a remote MDS is not being used.) Maximum number of TCPconnections is 2 (The default is 2 TCP connections being used, one for class F and the other for class 2 and 3.) Time Stamp is disabled OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 390
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] (The time stamp can be activated under the FCIP interface.) was issued, or since the last tunnel up.) 20-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 391
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] Verifying the Establishment of Default TCP error 0 FC CRC error, 0 iSCSI CRC error, 0 parity error OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 392
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Software Egress Counters 2312 good frames, 0 bad header cksum, 0 bad FIFO carries the FC Standard ELP. 20-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 393
[email protected] Figure 20-4 Second Capture of TCP Connection exchange link parameter Figure 20-5 shows the FC portion of the EISL initialization over the FCIP tunnel. Figure 20-5 Third Capture of TCP Connection Cisco OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 394
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] One-to-Three FCIP Tunnel Creation and Monitoring 10.10.11.0 255.255.255.0 interface gigabitethernet 2/1 20-20 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 395
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] Creating the FCIP Interface for the Second Tunnel Using the FCIP profile is bound to a physical port.) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-21 - Cisco MDS-9124 | Troubleshooting Guide - Page 396
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Listen Port is 3225 TCP parameters SACK is passive mode to guarantee MDS1 initiates a TCP connection.) 20-22 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 397
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] module-2# debug ips fcip fsm port 1 module-2# Mar 14 23:08:02 port1: 863:FCIP21: SUP-> Set Port mode 1 Mar 14 23:08:02 port1: 864:FCIP21: SUP-> Port VSAN (1) already set to same - Cisco MDS-9124 | Troubleshooting Guide - Page 398
20 Troubleshooting IP Storage Services Send documentation comments to [email protected] port1: 38:FCIP21: SUP-> Set Port mode 1 Mar 14 21:37:05 port1: 39:FCIP21: SUP-> Port VSAN (1) already frames 0 reass frames 20-24 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 399
Troubleshooting IP Storage Services FCIP Issues Send documentation comments to mdsfeedback-doc@cisco. no shut down on the FCIP interface.) Mar 14 21:32:27 port1: 8:FCIP21: SUP-> Set port VSAN: 1 Mar 14 21:32:27 port1: 9:FCIP21: Try to Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-25 - Cisco MDS-9124 | Troubleshooting Guide - Page 400
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Using Profile id 21 (interface GigabitEthernet2 (allowed active) (1-2) Trunk vsans (operational) (1-2) 20-26 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 401
20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] 23:49:06 port1: 1870:FCIP21: SUP-> Set Port mode 1 Mar 14 23:49:06 port1: 1871:FCIP21: SUP-> Port VSAN (1) already set to same Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-27 - Cisco MDS-9124 | Troubleshooting Guide - Page 402
Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Mar 15 00:01:48 port1: 3300:FCIP21: Send Link down to SUP Mar 15 00:01:48 port1: 3301:FCIP21: Start the bringup tunnel timer Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 403
Chapter 20 Troubleshooting IP Storage Services FCIP Issues Send documentation comments to [email protected] Mar 15 00:01:50 port1: 3309:FCIP21: Tunnel is not ADMIN UP state, reject new TCP connection from 10.10.10.2:65064 Mar 15 00:01:56 port1: 3310:FCIP21: SUP-> Set Port mode 1 Mar 15 00 - Cisco MDS-9124 | Troubleshooting Guide - Page 404
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Figure 20-7 shows a trace of time stamp difference failure. show an FCIP tunnel with a special frame. 20-30 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 405
Troubleshooting IP Storage Services FCIP Issues Send documentation comments to mdsfeedback-doc@cisco 15:25:38 port1: 857344:FCIP21: Send LINK UP to SUP Jan 14 15:25:39 port1: 857345:FCIP21: SUP-> Set trunk mode: 2 Jan 14 15:25:39 port1: Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-31 - Cisco MDS-9124 | Troubleshooting Guide - Page 406
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Step 4 vsan is 1 Trunk vsans (allowed active) (1-2) Trunk Trace of FCIP Tunnel with a Special Frame 20-32 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 407
Troubleshooting IP Storage Services FCIP Issues Send documentation comments to mdsfeedback-doc@cisco 14 15:14:30 port1: 855278:FCIP21: SUP-> Set Port mode 1 Jan 14 15:14:30 port1: 855279:FCIP21: SUP-> Port VSAN (1) already set to same Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-33 - Cisco MDS-9124 | Troubleshooting Guide - Page 408
FCIP Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Jan 14 15:14:31 port1: 855328:FCIP21: Bind the because the physical connection cannot handle the traffic. 20-34 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 409
Chapter 20 Troubleshooting IP Storage Services iSCSI Issues Send documentation comments to [email protected] To resolve the problem, tune the TCP parameters in the FCIP -10 Successful iSCSI Login Status Window OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-35 - Cisco MDS-9124 | Troubleshooting Guide - Page 410
iSCSI Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] On Solaris systems, a successful login is found in the /var/adm/ :not enabled telnet/ssh:not enabled 20-36 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 411
Chapter 20 Troubleshooting IP Storage Services iSCSI Issues Send documentation comments to [email protected] iscsi:not enabled authentication method:local username:iscsiuser secret:1234567812345678 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-37 - Cisco MDS-9124 | Troubleshooting Guide - Page 412
iSCSI Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] RADIUS Configuration Troubleshooting If iSCSI authentication is through client logs in successfully. 20-38 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 413
NAS port=1426 647 2003y3m14d 15h12m48s NAS port type=5 648 2003y3m14d 15h12m48s Service type=8 649 2003y3m14d 15h12m48s User (user002) authenticate OK. 650 2003y3m14d 654 2003y3m14d 15h12m54s NAS IP address=2887147911 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-39 - Cisco MDS-9124 | Troubleshooting Guide - Page 414
port type=5 670 2003y3m14d 15h12m54s Service type=8 671 2003y3m14d 15h12m54s User (user002) authenticate OK. Troubleshooting RADIUS Routing Configuration The switch sends authentication result for user002:user not present 20-40 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 415
Chapter 20 Troubleshooting IP Storage Services iSCSI Issues Send documentation comments to [email protected] Troubleshooting Dynamic iSCSI Configuration A physical Fibre Channel the logged-in iSCSI initiators. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-41 - Cisco MDS-9124 | Troubleshooting Guide - Page 416
20 Troubleshooting IP Storage Services Send documentation comments to [email protected] • Dynamic iSCSI configuration places all iSCSI initiators logging into the MDS 9000 switch iSCSI 2/7, Portal group tag: 0x86 20-42 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 417
Chapter 20 Troubleshooting IP Storage Services iSCSI Issues Send documentation comments to [email protected] VSAN ID 1, FCID 0x750105 show iscsi local-node Command these samples to help identify possible issues. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-43 - Cisco MDS-9124 | Troubleshooting Guide - Page 418
iSCSI Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] • show iscsi session detail • show iscsi stats • show iscsi , ExpCmdSN 1356, MaxCmdSN 1366, Barrier 0 20-44 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 419
Chapter 20 Troubleshooting IP Storage Services iSCSI Issues Send documentation comments to [email protected] MaxBurstSize 0, MaxConn 0, DataPDUInOrder No DataSeqInOrder No, InitialR2T 3816015476 Data-out bytes, 0 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-45 - Cisco MDS-9124 | Troubleshooting Guide - Page 420
iSCSI Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] fragments 1283382 packets output, 778114736 bytes 303109 Response :c5:26:0a (Seagate) scsi-fcp:target 20-46 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 421
Chapter 20 Troubleshooting IP Storage Services iSCSI Issues Send documentation comments to [email protected] Total number of entries = 5 show flogi database Command Output switch : Smoothed 165 ms, Variance: 35 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-47 - Cisco MDS-9124 | Troubleshooting Guide - Page 422
iSCSI TCP Performance Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Advertized window: Current: 61 KB, and the support for higher MTUs between the peers. 20-48 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 423
Troubleshooting IP Storage Services iSCSI TCP Performance Issues Send documentation comments to mdsfeedback-doc@cisco increase the MTU of the IP network between the peers. Jumbo support was enabled for the IPS ports, and the MTU for the Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-49 - Cisco MDS-9124 | Troubleshooting Guide - Page 424
iSCSI TCP Performance Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Any value starting with 64K (> 65535 = 0xFFFF module (in kilobytes).) tcp sack-enable 20-50 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 425
Module-Type Model Status 1 16 1/2 Gbps FC/Supervisor DS-X9216-K9-SUP active * 2 8 IP Storage Module DS-X9308-SMIP ok Mod Sw Hw World-Wide-Name(s) ( cisco:02.75af2f95624c.shark-nas iSCSI alias name: SHARK-NAS OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-51 - Cisco MDS-9124 | Troubleshooting Guide - Page 426
Troubleshooting IP Storage Services Send documentation comments to mdsfeedback-doc@cisco without jumbo frame support in the path. Without PMTU discovery, there will be problems.) Keepalive-timeout cisco:02.75af2f95624c.shark-nas iscsi tcp 20-52 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x - Cisco MDS-9124 | Troubleshooting Guide - Page 427
Chapter 20 Troubleshooting IP Storage Services iSCSI TCP Performance Issues Send documentation comments to [email protected] iSCSI Node name is iqn.1987-05.com.cisco:02.75af2f95624c.shark-nas See Figure 20-14.) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-53 - Cisco MDS-9124 | Troubleshooting Guide - Page 428
Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] Figure it can render the system unusable, requiring a reinstallation of the entire operating system. Only advanced users Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 429
Troubleshooting IP Storage Services iSCSI TCP Performance Issues Send documentation comments to mdsfeedback-doc@cisco the network in the middle supports jumbo frames. Use the show the TCP peer has problems acknowledging the data that Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-55 - Cisco MDS-9124 | Troubleshooting Guide - Page 430
iSCSI TCP Performance Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] 0 abort due to no memory, 0 duplicate Statistics: PDU: Command: 0, Response: 1612007 20-56 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 431
Troubleshooting IP Storage Services iSCSI TCP Performance Issues Send documentation comments to mdsfeedback-doc@cisco well as the host NIC) has to be capable of supporting this high MTU. If you do not have access to ... OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-57 - Cisco MDS-9124 | Troubleshooting Guide - Page 432
Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] iSLB Issues This section describes common troubleshooting issues for command to enable CFS distribution. 20-58 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 433
Chapter 20 Troubleshooting IP Storage Services iSLB Issues Send documentation comments to [email protected] iSCSI Initiator and Virtual Target Configuration Not Distributed ID from the initiator configuration. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-59 - Cisco MDS-9124 | Troubleshooting Guide - Page 434
error. To fix the problem, use another WWN or allow the system to assign one for the initiator using the static [nWWN | pWWN] system-assign command. iSLB Configuration, error. To fix the problem, use a different node name. 20-60 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 435
Chapter 20 Troubleshooting IP Storage Services iSLB Issues Send documentation comments to [email protected] iSLB Configuration Failed-"Pending iSLB CFS Config Has Reached command to get the status of the commit. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-61 - Cisco MDS-9124 | Troubleshooting Guide - Page 436
20 Troubleshooting IP Storage Services Send documentation comments to [email protected] fix the problem, use another WWN or allow the system to assign one for the initiator using the static [nWWN | pWWN] system-assign CLI Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 437
Chapter 20 Troubleshooting IP Storage Services iSLB Issues Send documentation comments to [email protected] iSLB Zones Not Present in Active Zone Set Symptom enabled. Use the islb zoneset activate CLI command. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-63 - Cisco MDS-9124 | Troubleshooting Guide - Page 438
iSLB Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] VRRP Master Overutilized Symptom VRRP master is that no zone or IVR zone changes are pending. 20-64 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 439
Chapter 20 Troubleshooting IP Storage Services iSLB Issues Send documentation comments to [email protected] iSLB CFS Commit Fails Symptom iSLB CFS commit fails config of the switch where the commit was performed. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 20-65 - Cisco MDS-9124 | Troubleshooting Guide - Page 440
iSLB Issues Chapter 20 Troubleshooting IP Storage Services Send documentation comments to [email protected] 20-66 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 441
, address, and port. IPv4 filters can also match on an ICMP type and type of service (ToS). This section includes the following topics: • Protocol Information, page 21-1 • Address Internet Control Message Protocol (ICMP). OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 442
do not support TCP or UDP port names. Table 21-1 Protocol UDP TCP and UDP Port Numbers for IPv4 Port dns tftp ntp radius accounting radius authentication snmp snmp-trap syslog Number 53 69 123 1646 or 1813 1645 or 1812 161 162 514 21-2 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 443
-reply unreachable redirect time exceeded traceroute Code 8 0 3 5 11 30 ToS Information IPv4 packets can be filtered based on the ToS conditions-delay, monetary-cost, normal-service, reliability, and throughput. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 444
Lists Send documentation comments to [email protected] Initial Troubleshooting Checklist Begin troubleshooting IP-ACLs by checking the following issues: Checklist Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that the access list has - Cisco MDS-9124 | Troubleshooting Guide - Page 445
Chapter 21 Troubleshooting IP Access Lists IP-ACL Issues Send documentation comments to [email protected] All Packets Are Blocked Symptom All packets are blocked. Table ACL name in the name field and click Add. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 446
IP-ACL Issues Chapter 21 Troubleshooting IP Access Lists Send documentation comments to [email protected] Step 6 Step 7 Step 8 Set the IP address, subnet mask, and cannot change the order of filters in an ACL. 21-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 447
Chapter 21 Troubleshooting IP Access Lists IP-ACL Issues Send documentation comments to [email protected] Step 4 Use the ip access-list or the ipv6 access-list creating IP-ACLs Using the CLI" section on page 21-6. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 21-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 448
IP-ACL Issues Chapter 21 Troubleshooting IP Access Lists Send documentation comments to [email protected] PortChannel Not Working with ACL Symptom PortChannel not ipv6 traffic-filter CLI command in interface mode. 21-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 449
in the integrated supervisor module. Refer to the Cisco MDS 9200 Series Hardware Installation Guide for more information on the Cisco MDS 9216i Switch. • The IPsec feature is not supported on the management interface. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 450
IPsec and IKE Algorithms for Microsoft Windows and Linux Platforms Table 22-2 lists the supported and verified settings for IPsec and IKE encryption authentication algorithms on the Microsoft Windows and Linux platforms. 22-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 451
signatures in digital certificates 768-bit DH 1024-bit DH 1536-bit DH Default Value 3DES SHA-1 Preshared keys 768-bit DH (1) OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 452
configure the authentication algorithm. Initial Troubleshooting Checklist Begin troubleshooting IPsec issues by checking the following issues: Checklist Check off Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that IKE has been configured - Cisco MDS-9124 | Troubleshooting Guide - Page 453
-8 • Verifying Interface Status Using Fabric Manager, page 22-9 • Verifying Interface Status Using the CLI, page 22-9 • Verifying Security Associations, page 22-12 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 454
IPsec Issues Chapter 22 Troubleshooting IPsec Send documentation comments to [email protected] • Security Associations Do Not Re-Key, page 22-15 • Clearing the crypto map in Step 1 are compatible on both switches. 22-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 455
switches. Ensure that the PFS settings in the show crypto map domain ipsec command outputs are configured the same on both switches. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 456
/255.255.255.255 Destination IP Address :10.10.100.231/255.255.255.255 Source port :*, Destination port :* Protocol * Physical port:0/1, Vlan_id:0/4095 22-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 457
tunnels are compatible. Verifying Interface Status Using the CLI To verify the status of the interfaces using the CLI, follow these steps: OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 458
IPsec Issues Chapter 22 Troubleshooting IPsec Send documentation comments to [email protected] Step 1 Issue the show interface gigabitethernet command on both protected by IPSec Write acceleration mode is off 22-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 459
connections is 2 Time Stamp is disabled QOS control code point is 0 QOS data code point is 0 B-port mode disabled TCP Connection Information OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 460
IPsec Issues Chapter 22 Troubleshooting IPsec Send documentation comments to [email protected] 2 Active TCP connections Control connection:Local 10.10.100.232:65492, .10.100.232, remote crypto endpt.:10.10.100.231 22-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 461
Chapter 22 Troubleshooting IPsec IPsec Issues Send documentation comments to [email protected] Step 2 mode:tunnel, crypto algo:esp-3des, auth algo:esp-md5- select 0 mask:0x0 Vlan 0 mask:0xfff Protocol 0 mask:0x0 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 462
1, 1970), remaining 206 4 secs Soft limit expiry 1100652397 secs (since January 1, 1970), remaining 204 2 secs Outbound MAC table index:125 Sequence number:7123 22-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 463
10 to 20 percent of the bytes are remaining in the bytes-based lifetime. To troubleshoot this problem, follow these steps: Step 1 Verify that traffic was flowing when the soft SA domain ipsec IPSec global statistics: OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 22-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 464
domain ipsec interface gigabitethernet 7/1 IPSec interface statistics: IKE transaction stats:0 num Inbound SA stats:1 num, 512 max Outbound SA stats:1 num, 512 max 22-16 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 465
, page 23-2 • Initial Troubleshooting Checklist, page 23-3 • SANTap Issues, page 23-6 Overview Note SANTap is not supported in Release 3.3.1. SANTap is an Intelligent Storage Services feature supported on the Storage Services Module (SSM). The SSM supports SANTap in Cisco MDS SAN-OS Release 2.0(2b - Cisco MDS-9124 | Troubleshooting Guide - Page 466
target that is included in SANTap-based services. The DVT is created in the host • SANTap does not support IVR. You cannot Cisco SAN-OS and SSI Release Prior to Cisco SAN-OS 3.2 Cisco SAN-OS 3.2 All releases Cisco SAN-OS 3.2 Prior to Cisco SAN-OS 3.2 23-2 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 467
Checklist Troubleshooting a SANTap problem involves gathering information about the configuration and connectivity of the various SANTap components. Begin your troubleshooting activity as follows: Checklist Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 468
SANTap Send documentation comments to [email protected] Common Troubleshooting Tools in Fabric Manager Use the following Fabric Manager procedures to verify the lun xmap id = 22:00:00:20:37:88:20:ef = 0x0 = 8 23-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 469
Chapter 23 Troubleshooting SANTap Initial Troubleshooting Checklist Send documentation comments to [email protected] dvt id dvt mode dvt vsan tgt pwwn tgt lun tgt vsan = 3 = 0 = 16 avt id = 12 tgt lun = 0x0 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 470
when troubleshooting SANTap problems. • Look for SSM_CRIT in Sup syslog messages. • SANTap logs are available via the show isapi tech-support CLI Host Login Problems, page 23-7 • ITL Problems, page 23-7 • Common Mismatch Problems, page 23-7 23-6 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 471
not supported by the interoperability matrix. The following lists common mismatch situations: Table 23-3 Common SANTap Mismatch Problems Problem version of the Sup and the SSI image are not SSI image. compatible. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 472
Chapter 23 Troubleshooting SANTap Send documentation comments to [email protected] Table 23-3 Common SANTap Mismatch Problems (continued) Problem Description Hosts, targets, HBAs or switches are not supported by the interoperability matrix. • Review the Cisco SANTap interoperability - Cisco MDS-9124 | Troubleshooting Guide - Page 473
Chapter 23 Troubleshooting SANTap SANTap Issues Send documentation comments to [email protected] Table 23-3 Common SANTap Mismatch Problems (continued) Problem Description A host is moved from the to the DVT. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 23-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 474
SANTap Issues Chapter 23 Troubleshooting SANTap Send documentation comments to [email protected] 23-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 475
. It includes the following sections: • Overview, page 24-1 • Initial Troubleshooting Checklist, page 24-3 • Digital Certificate Issues, page 24-4 Overview Public Key Infrastructure (PKI) support provides the means for the Cisco MDS 9000 Family of switches to obtain and use digital certificates for - Cisco MDS-9124 | Troubleshooting Guide - Page 476
by the issuing CA. CRLs and OCSP Support Two methods are supported for verifying that the peer certificate has not CRLs are published in a repository. Cisco MDS SAN-OS allows the manual configuration of pre-downloaded CRLs for the Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 477
area on the switch (bootflash). Cisco MDS SAN-OS supports certificate retrieval and enrollment using a manual cut-and-paste method. Cut-and Troubleshooting Tools in Fabric Manager Choose Switches > Security > PKI to access digital certificates. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 478
the challenge password in the Password field. Click Apply Changes. Or use the crypto ca enroll CLI command and enter a challenge password during enrollment. 24-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 479
RSA Key-Pair tab. b. Click Create Row and set the name and size field. c. Check the Exportable check box and click Create. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 480
, so you must remember this password. e. Click Apply Changes to save the changes. Request an identity certificate from the CA. Note The CA may require manual verification before issuing the identity certificate. 24-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 481
size: 1024 exportable: yes Vegas-1(config)# Step 5 Associate the RSA key pair to the trust point. Vegas-1(config)# crypto ca trustpoint myCA OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 482
your password will not be saved in the configuration. Please make a note of it. Password:nbv123 The subject name in the certificate will be: Vegas-1.cisco.com 24-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 483
manual Cisco/OU =netstorage/CN=Aparna CA serial=0A338EA1000000000074 notBefore=Nov 12 03:02:40 2005 GMT notAfter=Nov 12 03:12:40 2006 GMT MD5 Fingerprint=3D:33:62:3D:B4:D0:87:A0:70:DE:A3:87:B3:4E:24:BF purposes: sslserver sslclient ike OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 484
the RSA key pair that you want to delete and click Delete Row. Or use the no crypto key zeroize rsa CLI command 24-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 485
submode to remove the CA certificate from the trust point. switch(config)# crypto ca trustpoint myCA switch(config-trustpoint)# delete ca-certificate OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 24-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 486
import the certificates and RSA key pairs to the trust point. switch(config)# crypto ca import admin-ca pkcs12 bootflash:adminid.p12 nbv123 24-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 487
with the AutoNotify service. AutoNotify provides fast time to resolution of system problems by providing a direct notification path to Cisco customer support. The AutoNotify feature destination profiles of one or more types. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 488
An alert group is a predefined subset of Call Home alerts supported in all switches in the Cisco MDS 9000 Family. Different types of Call Home alerts are grouped into in the Cisco MDS 9000 Family System Messages Guide. 25-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 489
. Significant network impact. System is unusable. Critical conditions Troubleshooting Commands in the CLI The following commands may be useful in troubleshooting Call Home issues: • show callhome • show callhome user-def-cmds • callhome test OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 490
Call Home Send documentation comments to [email protected] Call Home Issues This section describes troubleshooting Call Home and includes the following topics: • Not it and remove the check, click it again. 25-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 491
for the user-defined profile (test1). Removes a previously configured urgency level and reverts it to the default of 0 (all messages are sent). OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 492
Home Contact Information Using Fabric Manager" section on page 25-7 or the "Configuring Call Home Contact Information Using the CLI" section on page 25-7. 25-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 493
to 128 alphanumeric characters are accepted in e-mail address format. Note You can use any valid e-mail address. You cannot use spaces. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 494
Call Home Issues Chapter 25 Troubleshooting Call Home Send documentation comments to [email protected] Receiving Too Many Call Home Alerts Symptom Receiving Level tab. Or use the show logging level CLI command. 25-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 495
Possible Cause Solution Inventory change occurred after the last The periodic inventory for Call Home is updated when the system reboot. switch restarts. Initiate a nondisruptive reboot to update inventory notification. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 25-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 496
Call Home Issues Chapter 25 Troubleshooting Call Home Send documentation comments to [email protected] 25-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 497
-depth health and configuration analysis tools leverage unique MDS 9000 switch capabilities including Fibre Channel ping and traceroute. Note You must have the same release of Fabric Manager Client and Fabric Manager Server. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 498
Passwords with Fabric Manager. Initial Troubleshooting Checklist Begin troubleshooting Fabric Manager issues by checking the following issues first: Checklist Checkoff Verify licensing requirements. See Cisco MDS 9000 Family Fabric Manager Configuration Guide. Verify that you have a compatible - Cisco MDS-9124 | Troubleshooting Guide - Page 499
user names configured. If you use the administrator password recovery procedure, you must manually reset the administrative password on the switch to resynchronize the SNMP and CLI user the Accelerate Discovery check box. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 500
Tips for Using Fabric Manager Chapter 26 Troubleshooting Fabric Manager Send documentation comments to [email protected] Red Line Through the Switch If a menu. Step 2 Select Layout -> Fix All Nodes from the menu. 26-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 501
steps: Step 1 Step 2 Step 3 Step 4 Go to the MDS 9000 folder. Edit the server.properties file with a text editor. Scroll until you find the line snmp.localaddress. If the line is commented, remove the comment character. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 502
Preferences from the Java WebStart Application Manager. Click the Manual radio button and enter the IP address of the field. Enter the HTTP port number used by your proxy service in the HTTP Port field. Click OK. Clearing Topology Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 503
fabric of Cisco MDS 9000 Family switches. Certain 2.x feature tabs will be disabled for any switches running a software version that does not support those features. Troubleshooting Fabric Manager Cache" section on page 26-8. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 504
this file is in C:\Program Files\Cisco Systems\MDS 9000 by default. Set web.allowDownload4All= or old versions of software. To correct this problem, clear Java cache. Step 1 Step 2 Cisco MDS 9000 Family Fabric Manager Configuration Guide. 26-8 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 505
Chapter 26 Troubleshooting Fabric Manager Troubleshooting Fabric Manager Web Client Send documentation comments to [email protected] Verifying Web Start. Refer to the installation instructions in the Cisco MDS 9000 Family Fabric Manager Configuration Guide, or the Release Notes for your - Cisco MDS-9124 | Troubleshooting Guide - Page 506
Fabric Manager Web Client Chapter 26 Troubleshooting Fabric Manager Send documentation comments to [email protected] Enabling the Java Runtime Console If you are URL or IP address appears in the Websites list. 26-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 507
to File > Preferences > Advanced and clear the applications folder or cache. You can manually delete the .javaws or cache directory. On Windows, this directory is under Documents and Settings : addUser OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 508
database size. This section contains the following topics: • Performance Manager Generates Java Error, page 26-13 • Performance Manager Not Working, page 26-13 26-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 509
password recovery procedure, you must manually reset the administrative password on the switch to resynchronize the SNMP and CLI user name and password. See the "Recovering the Administrator Password" section on page 2-32. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 26-13 - Cisco MDS-9124 | Troubleshooting Guide - Page 510
cache contains multiple Clear the Java Web Start cache. See Clearing Java Web versions of Device Manager software. Start Cache, page 26-11. 26-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 511
the show tech-support details CLI command. c. Use the tac-pac CLI command to redirect the output of the show tech-support details CLI command to a file, and then gzip the file. switch# tac-pac bootflash://showtech.switch1 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 512
the problem time? What troubleshooting steps have you attempted? Which, if any, of the following tools were used? - FC Analyzer, PAA-2, Ethereal, local or remote SPAN - CLI debug commands - FC traceroute, FC ping - Fabric Manager or Device Manager tools Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 513
them to your customer support representative: - show install all status - show system internal log install - show system internal log install details - show log nvram If your problem is related to Apply to copy the file. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x A-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 514
to an external system or pull files from an external system. File Server: 172.22.36.10 File to be copied to the switch: /etc/hosts The copy CLI command supports four transfer protocols the script use: run-script filename. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x A-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 515
switch. switch# system cores tftp://10.91.51.200/jsmith_cores switch# show system cores Cores are transferred to tftp://10.91.51.200/jsmith_cores Note The file name (indicated by jsmith_cores) must exist in the TFTP server directory. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 516
Using Core Dumps Appendix A Before Contacting Technical Support Send documentation comments to [email protected] Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x A-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 517
Failure Logging, page B-10 • Fabric Manager Tools, page B-13 • Fibre Channel Name Service, page B-18 • SNMP and RMON Support, page B-19 • Using RADIUS, page B-21 • Using Syslog, page B-21 • Using Fibre Channel SPAN, page B-23 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 518
no longer consolidated. CLI Debug The Cisco MDS 9000 Family switches support an extensive debugging feature set for actively troubleshooting a storage network. Using the CLI :40:26 flogi: next state [FLOGI_ST_PERFORM_CONFIG] Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-2 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 519
. Enables debugging for Call Home. Enables CDP debugging. Enables Cisco Fabric Services debugging. Enables CIM server debugging. Enables core daemon debugging. debugging. Enables kernel loadable module parameter debugging. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 520
debugging. Enables SVC debugging. Enables System debugging. Enables TL Port debugging. Fibre Channel ping and Fibre Channel traceroute features to troubleshoot problems with connectivity and path choices. Do not use basis. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-4 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 521
an end N port. FC ping uses the PRLI Extended Link Service, and verifies the presence of a Fibre Channel entity in traceroute are useful tools to check for network connectivity problems or verify the path taken toward a specific destination Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-5 - Cisco MDS-9124 | Troubleshooting Guide - Page 522
directly to the traced FCID (0xeb01e8) Latency 0 msec 2000000c306c2440(0xfffceb) -->idem, but looped around Latency 0 msec 2000000c30575ec0(0xfffced) --> first hop MDS on the return path from traced FCID to originor switch# Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-6 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 523
• The sum of all dynamically allocated memory that this process has received from the system; this includes memory that may have been returned to the system • The amount of CPU time the process has used, in microseconds OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 524
: • D = uninterruptible sleep (usually I/O). • R = runnable (on run queue). • S = sleeping. • T = traced or stopped. • Z = defunct ("zombie") process. • NR = not-running. • ER = should be running but currently not-running. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-8 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 525
Troubleshooting Tools and Methodology Using Cisco MDS 9000 Family Tools Send documentation comments to [email protected] Note The ER state typically designates a process that has been restarted too many times, causing the system Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 526
and cache are also included in the used memory statistics. Example B-6 show system resources Command switch# show system resources Load average: Processes : CPU states : Memory usage: 2340K buffers of failed cards. B-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 527
OBFL memory leak events. Enables the OBFL miscellaneous information. Enables the boot uptime, device version, and OBFL history. Disables all OBFL features. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 528
OBFL Log: Enabled Module: 6 OBFL Log: error-stats exception-log miscellaneous-error obfl-log (boot-uptime/device-version/obfl-history) system-health stack-trace Enabled Enabled Enabled Enabled Enabled Enabled Enabled B-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 529
stack-trace show logging onboard starttime show logging onboard system-health Purpose Displays the boot and uptime information. about using Cisco Fabric Manager, refer to the Cisco MDS 9000 Family Fabric Manager Configuration Guide. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 530
Figure B-2 Cisco Device Manager Summary View The Summary View window lets you analyze switch performance issues, diagnose problems, and change parameters to resolve problems or of the components of a specific switch. B-14 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 531
Troubleshooting Tools and Methodology Using Cisco MDS 9000 Family Tools Send documentation comments to [email protected] Figure B-3 Switch Health Analysis Window The Switch Health Analysis window displays any problems -05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-15 - Cisco MDS-9124 | Troubleshooting Guide - Page 532
tool for troubleshooting problems that occur when merging zones configured on different switches. Select the Zone Merge option on the Fabric Manager Tools menu to determine if two connected switches have compatible zone configurations. B-16 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 533
provides a set of preconfigured traps and informs that are automatically generated and sent to the destinations (trap receivers) chosen by the user. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-17 - Cisco MDS-9124 | Troubleshooting Guide - Page 534
register themselves with the name service, which is then distributed among all participating fabric switches. This information can then be used to help determine the identity and topology of nodes connected to the fabric. B-18 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 535
Cisco MDS 9000 Family switches. SNMPv3 provides extended security. Each switch can be selectively enabled or disabled for SNMP service. In addition, each switch can be configured with a method of handling SNMPv1 and v2 requests. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release - Cisco MDS-9124 | Troubleshooting Guide - Page 536
B Troubleshooting Tools and Methodology Send documentation comments to [email protected] Note During initial configuration of your switch, the system prompts you to define SNMP v1 or V2 community strings and to create a SNMP v3 username and password. Cisco MDS 9000 Family switches support - Cisco MDS-9124 | Troubleshooting Guide - Page 537
can limit the severity levels that are reported for specific services within the switch. For example, you may wish only to report debug events for the FSPF service but record all severity level events for the Zoning service. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-21 - Cisco MDS-9124 | Troubleshooting Guide - Page 538
Using Cisco MDS 9000 Family Tools Appendix B Troubleshooting Tools and Methodology Send documentation comments to [email protected] A unique feature within the Cisco MDS 9000 Family switches is the ability to send RADIUS accounting records to the Syslog service. The advantage of this - Cisco MDS-9124 | Troubleshooting Guide - Page 539
which disrupts input and output (I/O) to and from the device. This problem is worse when the point of analysis is on an Inter-Switch Link session. When you enter this command, the system displays a submenu, which lets you configure -05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-23 - Cisco MDS-9124 | Troubleshooting Guide - Page 540
Cisco MDS 9000 Family Configuration Guide. Using Cisco Network Management Products This section describes network management tools that are available from Cisco and are useful for troubleshooting problems with Cisco MDS The Cisco MDS 9000 Family Port Analyzer Adapter does not support half-duplex - Cisco MDS-9124 | Troubleshooting Guide - Page 541
For more information on using the Cisco MDS 9000 Family Port Analyzer Adapter see the Cisco MDS 9000 Family Port Analyzer Adapter Installation and Configuration Guide. Cisco Fabric Analyzer The ultimate tool for troubleshooting network protocol problems is the protocol analyzer. Protocol analyzers - Cisco MDS-9124 | Troubleshooting Guide - Page 542
frames. This capability provides flexibility for troubleshooting problems in remote locations. The Cisco Fabric Analyzer captures and analyzes control traffic 01 Detection Function Fabric Detected (0x00000001) Num Entries 1 B-26 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 543
the system. supported on the 8-port IP Storage Services (IPS-8) module and on the 4-port IP Storage Services Cisco MDS 9000 Family CLI Configuration Guide. Using Other Troubleshooting Products This section describes products from other vendors that you might find useful when troubleshooting problems - Cisco MDS-9124 | Troubleshooting Guide - Page 544
are not necessarily aware of problems occurring at the Fibre Channel system, you can use the Diskmon or Disk Management tool to verify accessibility of the storage and to perform some basic monitoring and administrative tasks on the visible volumes. B-28 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 545
, or a third-party utility like Extreme SCSI. Every UNIX version provides similar utilities, but this guide only provides examples for Solaris. Refer to the documentation for your specific operating system for details. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x B-29 - Cisco MDS-9124 | Troubleshooting Guide - Page 546
Using Host Diagnostic Tools Appendix B Troubleshooting Tools and Methodology Send documentation comments to [email protected] B-30 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 547
per switch (includes all VSANs). Zone sets 500 zone sets per switch (includes all VSANs). Supported hops for all major storage, server, and HBA vendors 7 hops (diameter of the SAN fabric (includes all VSANs). 12 hops. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x C-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 548
you have the large Fabric design validated by Cisco Advanced Services. 2. This is the number of trunking-enabled ISL ports multiplied by the number of VSANs in the switch. 3. SANTap is not supported in SAN-OS Release 3.3(1). Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x C-2 OL-9285-05 - Cisco MDS-9124 | Troubleshooting Guide - Page 549
verifying with CLI (procedure) 7-3 verifying with Fabric Manager (procedure) 7-3 Cisco Fabric Services. See CFS CLI common troubleshooting commands 1-4 debug commands B-2 clock modules 4-12 commit fails device alias 15-3, 15-11 SDV 12-3 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-1 - Cisco MDS-9124 | Troubleshooting Guide - Page 550
in-band management 16-1 CUP in-band management description 16-6 customer support, collecting information A-1 D destination profiles configuring 25-1 device alias commands VSAN limitations 15-3 Device Manager IN-2 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x password issues 26-13 port - Cisco MDS-9124 | Troubleshooting Guide - Page 551
checking in Fabric Manager Server license 6-10 map layout 26-4 problems 26-2 recommened JRE version (table) 2-5 troubleshooting tools 1-3 using over FCIP 26-5 using with multiple NICs 16-5 configuring 16-1 CUP in-band management 16-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-3 - Cisco MDS-9124 | Troubleshooting Guide - Page 552
overview 4-1 startup issues 4-2 IN-4 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x troubleshooting 4-13 I IKE allowed transforms ( statistics 22-15 supported platforms (table) 22-3 troubleshooting 22-5 verifying configuration 22-6, 22-7 IP security. See IPsec IP services 20-5 iSCSI OL- - Cisco MDS-9124 | Troubleshooting Guide - Page 553
support (table) 13-5 restriction 23-2 restrictions 13-2 OL-9285-05 SDV limitations 12-2 traffic blocked 13-11 transit VSANs 13-2 troubleshooting 6-3 displaying with Fabric Manager Web Services 6-4 feature-based 6-1 grace period warnings. 6-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x - Cisco MDS-9124 | Troubleshooting Guide - Page 554
CLI commands 9-2 core switch 9-6 end device login 9-4 initial checklist 9-2 NPIV 9-5 restrictions 9-2 VSAN mismatches 9-5 IN-6 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x O OBFL configuring for a module B-12 configuring for the switch B-11 description B-10 displaying configuration - Cisco MDS-9124 | Troubleshooting Guide - Page 555
) 17-6 related documents i-xxx RMON B-17 roles, admin 2-34 S SAN registration 1-5 SANTap acronyms 23-2 CLI commands 23-4 Fabric Manager tools 23-4 host login problems 23-7 initial checklist 23-3 interface restrictions 23-3 OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-7 - Cisco MDS-9124 | Troubleshooting Guide - Page 556
selecting for supervisor modules 5-13 IN-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Software Installation Wizard ( 4-17 troubleshooting 4-14 switches maximum numbers C-1 switch health analysis B-14 syslog See system messages system health failure 2-33 system images selecting - Cisco MDS-9124 | Troubleshooting Guide - Page 557
documentation comments to [email protected] system messages overview 1-10, B-21 problems 4-13 IP services 20-5 iSCSI issues 20-35 modules 4-21 overview 1-3 power supplies 4-7 SSM recovery 3-8 switching and services OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-9 - Cisco MDS-9124 | Troubleshooting Guide - Page 558
documentation comments to [email protected] activation fails 12-6 zones troubleshooting checklist 14-1 troubleshooting with CLI 14-2 troubleshooting with Fabric Manager 14-2 zone set activation 14-8 zone sets maximum number in a switch C-1 IN-10 Cisco MDS 9000 Family Troubleshooting Guide - Cisco MDS-9124 | Troubleshooting Guide - Page 559
Index Send documentation comments to [email protected] OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x IN-11 - Cisco MDS-9124 | Troubleshooting Guide - Page 560
Index Send documentation comments to [email protected] IN-12 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05
-
1 -
2 -
3 -
4 -
5 -
6 -
7 -
8
-
9
-
10
-
11
-
12
-
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
-
21
-
22
-
23
-
24
-
25
-
26
-
27
-
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
-
37
-
38
-
39
-
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
-
48
-
49
-
50
-
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
-
60
-
61
-
62
-
63
-
64
-
65
-
66
-
67
-
68
-
69
-
70
-
71
-
72
-
73
-
74
-
75
-
76
-
77
-
78
-
79
-
80
-
81
-
82
-
83
-
84
-
85
-
86
-
87
-
88
-
89
-
90
-
91
-
92
-
93
-
94
-
95
-
96
-
97
-
98
-
99
-
100
-
101
-
102
-
103
-
104
-
105
-
106
-
107
-
108
-
109
-
110
-
111
-
112
-
113
-
114
-
115
-
116
-
117
-
118
-
119
-
120
-
121
-
122
-
123
-
124
-
125
-
126
-
127
-
128
-
129
-
130
-
131
-
132
-
133
-
134
-
135
-
136
-
137
-
138
-
139
-
140
-
141
-
142
-
143
-
144
-
145
-
146
-
147
-
148
-
149
-
150
-
151
-
152
-
153
-
154
-
155
-
156
-
157
-
158
-
159
-
160
-
161
-
162
-
163
-
164
-
165
-
166
-
167
-
168
-
169
-
170
-
171
-
172
-
173
-
174
-
175
-
176
-
177
-
178
-
179
-
180
-
181
-
182
-
183
-
184
-
185
-
186
-
187
-
188
-
189
-
190
-
191
-
192
-
193
-
194
-
195
-
196
-
197
-
198
-
199
-
200
-
201
-
202
-
203
-
204
-
205
-
206
-
207
-
208
-
209
-
210
-
211
-
212
-
213
-
214
-
215
-
216
-
217
-
218
-
219
-
220
-
221
-
222
-
223
-
224
-
225
-
226
-
227
-
228
-
229
-
230
-
231
-
232
-
233
-
234
-
235
-
236
-
237
-
238
-
239
-
240
-
241
-
242
-
243
-
244
-
245
-
246
-
247
-
248
-
249
-
250
-
251
-
252
-
253
-
254
-
255
-
256
-
257
-
258
-
259
-
260
-
261
-
262
-
263
-
264
-
265
-
266
-
267
-
268
-
269
-
270
-
271
-
272
-
273
-
274
-
275
-
276
-
277
-
278
-
279
-
280
-
281
-
282
-
283
-
284
-
285
-
286
-
287
-
288
-
289
-
290
-
291
-
292
-
293
-
294
-
295
-
296
-
297
-
298
-
299
-
300
-
301
-
302
-
303
-
304
-
305
-
306
-
307
-
308
-
309
-
310
-
311
-
312
-
313
-
314
-
315
-
316
-
317
-
318
-
319
-
320
-
321
-
322
-
323
-
324
-
325
-
326
-
327
-
328
-
329
-
330
-
331
-
332
-
333
-
334
-
335
-
336
-
337
-
338
-
339
-
340
-
341
-
342
-
343
-
344
-
345
-
346
-
347
-
348
-
349
-
350
-
351
-
352
-
353
-
354
-
355
-
356
-
357
-
358
-
359
-
360
-
361
-
362
-
363
-
364
-
365
-
366
-
367
-
368
-
369
-
370
-
371
-
372
-
373
-
374
-
375
-
376
-
377
-
378
-
379
-
380
-
381
-
382
-
383
-
384
-
385
-
386
-
387
-
388
-
389
-
390
-
391
-
392
-
393
-
394
-
395
-
396
-
397
-
398
-
399
-
400
-
401
-
402
-
403
-
404
-
405
-
406
-
407
-
408
-
409
-
410
-
411
-
412
-
413
-
414
-
415
-
416
-
417
-
418
-
419
-
420
-
421
-
422
-
423
-
424
-
425
-
426
-
427
-
428
-
429
-
430
-
431
-
432
-
433
-
434
-
435
-
436
-
437
-
438
-
439
-
440
-
441
-
442
-
443
-
444
-
445
-
446
-
447
-
448
-
449
-
450
-
451
-
452
-
453
-
454
-
455
-
456
-
457
-
458
-
459
-
460
-
461
-
462
-
463
-
464
-
465
-
466
-
467
-
468
-
469
-
470
-
471
-
472
-
473
-
474
-
475
-
476
-
477
-
478
-
479
-
480
-
481
-
482
-
483
-
484
-
485
-
486
-
487
-
488
-
489
-
490
-
491
-
492
-
493
-
494
-
495
-
496
-
497
-
498
-
499
-
500
-
501
-
502
-
503
-
504
-
505
-
506
-
507
-
508
-
509
-
510
-
511
-
512
-
513
-
514
-
515
-
516
-
517
-
518
-
519
-
520
-
521
-
522
-
523
-
524
-
525
-
526
-
527
-
528
-
529
-
530
-
531
-
532
-
533
-
534
-
535
-
536
-
537
-
538
-
539
-
540
-
541
-
542
-
543
-
544
-
545
-
546
-
547
-
548
-
549
-
550
-
551
-
552
-
553
-
554
-
555
-
556
-
557
-
558
-
559
-
560
 |
 |
Send documentation comments to [email protected]
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Cisco MDS 9000 Family
Troubleshooting Guide, Release 3.x
Cisco MDS SAN-OS for Release 3.3(1)
May 2008
Text Part Number: OL-9285-05