Dell PowerEdge M610 Fabric OS Troubleshooting and Diagnostics Guide
Dell PowerEdge M610 Manual
View all Dell PowerEdge M610 manuals
Add to My Manuals
Save this manual to your list of manuals |
Dell PowerEdge M610 manual content summary:
- Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 1
53-1001769-01 ® 30 March 2010 Fabric OS Troubleshooting and Diagnostics Guide Supporting Fabric OS v6.4.0 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 2
www.brocade.com/support/oscd. Brocade Document History Title Fabric OS Troubleshooting and Diagnostics Guide Fabric OS Troubleshooting and Diagnostics Guide Publication number 53-0000853-01 53-1001187-01 Fabric OS Troubleshooting and 53-1001340-01 Diagnostics Guide Fabric OS Troubleshooting - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 3
Getting technical help xiii Document feedback xiv Chapter 1 Introduction to Troubleshooting In this chapter 1 Troubleshooting overview 1 Network time protocol 1 Most common problem areas 2 Questions for common symptoms 2 Gathering information for your switch support provider 5 Setting up - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 4
considerations 41 Supported platforms 42 Maximum mirror connections 42 Configuring a port to be a mirror port 43 Adding a port mirror connection 44 Deleting a port mirror connection 44 Displaying port mirror connections 45 iv Fabric OS Troubleshooting and Diagnostics Guide 53-1001769 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 5
In this chapter 53 Blade troubleshooting tips 53 Firmware download issues 54 Troubleshooting firmwareDownload 57 Gathering additional issues 66 Virtual Fabrics In this chapter 69 General Virtual Fabric troubleshooting 69 Fabric identification issues 70 Logical Fabric issues 70 Base switch - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 6
for zoning problems 83 Gathering additional information 84 Working With Diagnostic Features In this chapter 85 About Fabric OS diagnostics 85 Diagnostic information 85 Power-on self summary of port errors for a switch 96 vi Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 7
and power supply 97 Checking the status of the fans 97 Checking the status of a power supply 98 server 104 Saving comprehensive diagnostic files to the server 104 Switch Type and Blade ID Hexadecimal Example of hexadecimal triplet conversion 109 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 8
viii Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 9
•Supported hardware and software x •What's new in this document x •Document conventions xi •Additional information xiii •Getting technical help xiii •Document feedback xiv How this document is organized The document contains the following chapters: • Chapter 1, "Introduction to Troubleshooting - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 10
hardware and software In those instances in which procedures or parts of procedures documented here apply to some switches but not to others, this guide identifies exactly which switches are supported and which are not. Although many different software and hardware configurations are tested - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 11
supported in the current version of firmware. - The iSCSI chapter has been removed and put in the iSCSI Administrator's Guide. For further information about documentation , this manual specifically notes those cases in which a command is case sensitive. Command syntax in this manual follows these - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 12
in this manual. They firmware, specific to Brocade and Fibre Channel, see the Brocade Glossary. For definitions of SAN-specific terms, visit the Storage Networking Industry Association online dictionary at: http://www.snia.org/education/dictionary xii Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 13
products/index.page For additional Brocade documentation, visit the Brocade Web site support supplier for hardware, firmware, and software support problem, including the switch or fabric behavior immediately following the problem, and specific questions Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 14
next to the power supply bays • [email protected] Provide the title and version number of the document and as much detail as possible about your comment, including the topic heading and page number and your suggestions for improvement. xiv Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 15
1 In this chapter •Troubleshooting overview 1 •Most common problem areas 2 •Questions for common symptoms 2 •Gathering information for your switch support provider 5 •Building a case for your switch support provider 7 Troubleshooting overview This book is a companion guide to be used in - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 16
possible areas to check. You may notice that an intermittent connectivity problem has lots of variables to look into, such as the type of connection between the two devices, how the connection is behaving, and the port type involved. 2 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 17
to check Chapter or Document Firmware or application download Hardware connections Chapter 2, "General Issues" Chapter 5, "FirmwareDownload Errors" Chapter 7, "Virtual Fabrics" Firmware or application download Chapter 5, "FirmwareDownload Errors" FTP or SCP server or USB availability Chapter - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 18
No light on LEDs Performance problems Licensing Fabric parameters Segmentation Virtual QOS configuration mismatch Password recovery Chapter or Document Chapter 3, "Connectivity Issues" Chapter 2, " Chapter 6, "Security Issues" 4 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 19
or Document Chapter 6, "Security Issues" Chapter 7, "Virtual Fabrics" Chapter 9, "Zone Issues" Chapter 9, "Zone Issues" Chapter 9, "Zone Issues" Gathering information for your switch support provider If you are troubleshooting a production system, you must gather data quickly. As soon as a problem - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 20
information for your switch support provider 2. Type the appropriate supportSave command based on your needs: • If you are saving to an FTP or SCP server, use the following syntax switch using an account assigned to the admin role. 6 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 21
command to determine the answer. • Do you have encryption blades or switches installed in the fabric? Yes or no. • Do you have Virtual Fabrics enabled in the fabric? Yes or no. Use the switchShow command to determine the answer. Fabric OS Troubleshooting and Diagnostics Guide 7 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 22
as possible prior to contacting the SAN technical support vendor. Document the sequence of events by answering the following questions: • When did problem occur? • Is this a new installation? • How long has the problem been occurring? • Are specific devices affected? If so, what are their World - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 23
you to gather additional information. The additional information is necessary in order for your switch support provider to effectively and efficiently troubleshoot your issue. Refer to the chapter or document specified for the commands whose data you need to capture: • Configurations, see Chapter - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 24
1 Building a case for your switch support provider 10 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 25
so you cannot apply one license to different switches. Before calling your switch support provider, verify that you have the correct licenses installed by using the licenseShow NTP, refer to the Fabric OS Administrator's Guide. Fabric OS Troubleshooting and Diagnostics Guide 11 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 26
review of the RAS logs is necessary to prevent minor problems from becoming larger issues, or in catching problems at an early stage. Below are some common problems Administrator's Guide for Server or can prevent a switch from joining the fabric. MQ errors are rare and difficult to troubleshoot - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 27
blades; the specific item is listed in the error message. See the Fabric OS Message Reference for information specific minimal supportSave can be collected and sent to your service support provider for analysis. Not every reboot activates the Troubleshooting and Diagnostics Guide 13 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 28
may be caused by one of the following: - Power-cycle of the switch or CP. - Linux reboot command is unable to load the firmware to a usable state. HASM log contains more detail and specific information on this type support USB. 14 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 29
available on AP blades. • If contact you service support provider Service enables you to route the ECHO generated when an fcPing command is issued on a switch, providing fcPing capability between two devices in different fabrics across the FC router. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 30
and destination. In addition, two Fibre Channel Extended Link Service (ELS) requests will be generated. The first ELS request of devices. ATTENTION There are some devices that do not support the ELS ECHO request. In these cases, the device will Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 31
investigate these ISLs to determine the exact links. ATTENTION There are some devices that do not support the ELS ECHO request. In these cases, the device will either not respond to the request to aid in designing fabric redundancy. Fabric OS Troubleshooting and Diagnostics Guide 17 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 32
2 FC-FC routing connectivity • Determines the specific ISLs and internal links with failures. • Exercises all ISL links in the base fabric for a logical further investigate these ISLs to determine the exact links giving the errors. 18 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 33
Fabric OS v6.3.0 or later. • Superping requires that the FC Echo ELS frame is supported by end-devices. • In TI Zones, when failover is disabled and superping is executed Zone. • This feature is not supported in interopMode 2 or 3. Fabric OS Troubleshooting and Diagnostics Guide 19 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 34
is not accepted. Use the pathInfo command to display basic path information to a specific domain in command line mode: switch:admin> pathinfo 91 Target port is Embedded path (yes, y, no, n): [no] Source route (yes, y, no, n): [no] 20 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 35
resources to process them. Use the enforce tag feature to prevent a backbone switch from accepting unwanted LSAN zone databases into its local database. Fabric OS Troubleshooting and Diagnostics Guide 21 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 36
the host and import them faster. This helps sensitive hosts to do a quick discovery without timing out or cause an application failure. 22 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 37
•Link issues 25 •Connection problems 25 •Link failures 28 • a specific source port and destination port traffic passing though any switch port. This is only supported between services or routing topology information from one edge fabric to another. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 38
this process can help you determine where a problem resides. For example, if your switch cannot form Server, use the nsShow or nsAllShow command. For more information about these name server commands, refer to Fabric OS Command Reference. 24 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 39
LED or power status LED, refer to that switch's model hardware reference manual. There is problems Determine if the problem is the target or the host, then continue to divide the suspected problem-path in half until you can pinpoint the problem Troubleshooting and Diagnostics Guide 25 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 40
failure" on page 29 as the next potential trouble spot. Checking the Name Server (NS) 1. Enter the nsShow command on the switch to determine if the device is attached: switch:admin> nsshow The Local Name Server has 9 entries { 26 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 41
. There may be a time-out communication problem between edge devices and the name server, or there may be a login issue. First check the edge device documentation to determine if there is a time-out and the switch probed the device. Fabric OS Troubleshooting and Diagnostics Guide 27 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 42
3 Link failures Link failures A link failure occurs when a server, storage, or switch device is connected to a switch, but the link 0 - Auto Negotiate 1 - 1Gbps 2 - 2Gbps 4 - 4Gbps 8 - 8Gbps ax - Auto Negotiate + enhanced retries 28 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 43
on the switch; or if you are looking for a specific port, enter the portLogDumpPort command. b. Check argument number of the loop initialization. This behavior sometimes causes trouble with old HBAs. Checking for a point-to-point Fabric OS Troubleshooting and Diagnostics Guide 29 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 44
is another switch, the port becomes an E_Port. If there is a problem with the host or target device, enter the portCfgGPort to force the to read a portlog dump, contact your switch support provider for assistance. Marginal links A marginal link Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 45
with a known good working SFP. If you see this issue on an 8 Gbps blade, use the portCfgFillWord command to reduce EMI. • The crc_g_eof counter are frames with CRC . One problem this could be related to is ISL oversubscription. Fabric OS Troubleshooting and Diagnostics Guide 31 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 46
, the port is bad. Replace the port blade or switch. • If the loopback test did problem still exists. • If the portErrShow output displays a normal number of generated errors, the issue is solved. • If the portErrShow output still displays a high number of generated errors, follow the troubleshooting - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 47
" (Trunk master) Pinpointing problems with device logins 1. Log in to the switch as admin. 2. Enter the switchShow command; then, check for state logins. 3. Enter the portCfgShow command to see if the port is configured correctly. Fabric OS Troubleshooting and Diagnostics Guide 33 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 48
the port errors every five or ten minutes until you see the problem occur again. NOTE When two shared ports on an FC4-48 blade are receiving traffic and the primary port goes offline, all the 13 Offline No_Module PRESENT U_PORT LED 34 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 49
basic testing of the switch circuit. If a structural test fails, replace the main board or port blade. • Functional tests verify the intended operational behavior of the switch by running frames through ports or bypass circuitry. Fabric OS Troubleshooting and Diagnostics Guide 35 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 50
--linkbeacon command can be used to beacon a target port on the switch. For more information on using this command, refer to the Brocade Adapters Administrator's Guide. 36 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 51
and 8 Gbps ASIC is using the Turbo-Ram BIST circuitry. This allows the BIST controller to such as the BB credit, can be changed for specific applications or operating environments, but must be the same Blade ID".) • BB credit • PID format Fabric OS Troubleshooting and Diagnostics Guide 37 53 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 52
main fabric. For details on uploading and downloading configurations, see the Fabric OS Administrator's Guide. Reconciling a domain ID conflict If a domain ID conflict appears, the conflict is only IDs for each of those switches. 38 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 53
Transmission Mode (yes, y, no, n): [no] Arbitrated Loop parameters (yes, y, no, n): [no] System services (yes, y, no, n): [no] Portlog events enable (yes, y, no, n): [no] ssl attributes (yes The port level zoning may be affected Fabric OS Troubleshooting and Diagnostics Guide 39 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 54
's Guide for more information on that specific software feature. Port mirroring With port mirroring, you can configure a switch port to mirror the traffic between a specific source and destination port. This is only supported between F_Ports. This is a useful way to troubleshoot a problem port - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 55
, and the FA4-18, FC10-6, FC4-16IP, FC4-48, and FR4-18i blades have the following considerations: • If you define multiple mirror connections for the same F_PORT (SID or DID), all the connections must share the same mirror port. Fabric OS Troubleshooting and Diagnostics Guide 41 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 56
platform or blade supporting port mirroring supports both the mirror ports and the mirror connections. Exemptions are listed in Table 8. TABLE 8 Port Mirroring platform supportability Brocade 3 Brocade 48000 1 Brocade 4900 3 42 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 57
number/] --enable command. NOTE The enable command enables the port as a mirror port. The disable command disables the mirror port configuration. Fabric OS Troubleshooting and Diagnostics Guide 43 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 58
DestID command. For example, to delete the port mirror connection on mirror port 2, you might type: switch:admin> portMirror --del 0x011400 0x240400 44 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 59
mirror connection(s) configured: 4 Mirror_Port SID DID State 18 0x070400 0x0718e2 Enabled 18 0x070400 0x0718e3 Enabled 18 0x070400 0x0718ef Enabled 18 0x070400 0x0718e0 Enabled Fabric OS Troubleshooting and Diagnostics Guide 45 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 60
3 Port mirroring 46 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 61
Aug 15 15:13 support\ 106MB 2007 Aug 24 05:36 support1034\ 105MB 2007 Aug 23 06:11 config\ 0B 2007 Aug 15 15:13 firmware\ 380MB 2007 Aug 15 15:13 FW_v6.0.0\ 380MB 2007 Aug 15 15:13 Available space on usbstorage 74% Fabric OS Troubleshooting and Diagnostics Guide 47 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 62
to Services FTP service. User (10.255.252.50:(none)): userFoo 331 Please specify the password. Password: 530 Login incorrect. Login failed. If your login to the FTP or SCP server has failed, verify the username and password are correct. 48 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 63
corrected the problems will help you to avoid this problem in future the FTP server. • The USB path is incorrect. If your platform supports a USB memory device, . You can continue with the instructions. Otherwise issue the command again as Troubleshooting and Diagnostics Guide 49 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 64
from your computer when you are analyzing the problem. Send this and all logs to your switch support provider. Messages captured in the logs Configuration configuration without disabling the switch was unsuccessful. (Audit log) 50 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 65
In the hardware reference manuals for the Brocade 48000, DCX, and DCX-4S modular switches there is a guide for FC port setting tables. The tables can be used to record configuration information for the various blades. Troubleshooting and Diagnostics Guide 51 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 66
4 Brocade configuration form 52 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 67
In this chapter •Blade troubleshooting tips 53 •Firmware download issues 54 •Troubleshooting firmwareDownload 57 •USB error handling 57 •Considerations for downgrading firmware 58 Blade troubleshooting tips This chapter refers to the following specific types of blades inserted into either the - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 68
service provider. Do not disconnect the switch from power during the process because the switch could become inoperable when rebooted. Symptom The following symptoms describe common firmware after the firmwareCommit is complete. 54 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 69
74% Example of error message switch:admin> firmwaredownload Server Name or IP Address: 192.126.168.115 User Name: userFoo File Name: /users/home/userFoo/firmware/v6.4.0 Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 2 Fabric OS Troubleshooting and Diagnostics Guide 55 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 70
firmware/v6.4.0 Network Protocol(1-auto-select, 2-FTP, 3-SCP) [1]: 2 Password: Server IP: 192.126.168.115, Protocol IPv4 Checking system settings for firmwaredownload... Sanity check failed because firmwaredownload is already in progress. 56 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 71
remedy this. Verify blade is ENABLED. (error 3) Gathering additional information You should follow these best practices for firmware download before you start under specific scenarios and details what actions you should take after the error occurs. Fabric OS Troubleshooting and Diagnostics Guide 57 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 72
" accordingly. This example shows hardware-related messages for the same downgrade example: ecp:admin> firmwaredownload Type of Firmware (FOS, SAS, or any application) [FOS]: Server Name or IP Address: 10.1.2.3 User Name: userfoo 58 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 73
because a range of ports not supported by the target firmware are assigned to Logical Switches instead firmware version v6.4.0 or higher. Please switch to standard compression using the "portCfg fciptunnel modify" command before downgrading. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 74
FX8-24 AP blades (blade ID 75) in the system. Brocade DCX Extension blades are not supported on firmware v6.2.0 or earlier, so the firmware download operation failed. Power off or remove the blade and try the operation again. 60 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 75
the SAS image while the blade was operational. Retry the firmwaredownload command again. Platform These messages are switch features or fabric-wide settings that need to be removed or disabled before downgrading the firmware. Fabric OS Troubleshooting and Diagnostics Guide 61 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 76
logical switches, delete the base switch, and disable Virtual Fabrics prior to downgrading the firmware. Message Routing This message refers to any route settings that need to be changed reset the IOD delay to its default value. 62 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 77
and factory. If you do not know the root password, you need to contact your service support provider to recover admin passwords. Recovering passwords 1. Open a CLI connection (serial or Telnet , contact your switch service provider. Fabric OS Troubleshooting and Diagnostics Guide 63 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 78
command to set all passwords to default. How to recover boot PROM password? Contact your switch service provider and provide the recovery string. How do I recover a user, admin, or factory password mode on the switch to ON. 64 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 79
, the Fibre Channel driver rejects all other ELS problem area, collect a supportSave -n from all switches in the fabric. • If you think it may be related to E_Port authentication then collect a supportSave -n from both switches of the affected E_Port. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 80
server, check that the routing tables are set up correctly for your network. If you continue to have problems, collect the data in the next section and contact your switch support for both power-up and service provider for repair. 66 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 81
FIPS issues 6 Fabric OS Troubleshooting and Diagnostics Guide 67 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 82
6 FIPS issues 68 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 83
be manually changed supported. • A logical fabric cannot have EX_Ports using extended ISLs and cannot serve as a backbone to any EX_Port traffic. Similarly, the default switch cannot be part of a fabric that serves as a backbone to any EX_Port traffic. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 84
on. Although VE_Ports are allowed in a base switch, Fabric OS v6.2.0 and later do not support the use of VE_Ports to carry traffic for logical fabrics using XISLs. They can be used to carry using the lfCfg --lislenable command. 70 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 85
indicates the specified switch in the logical fabric using XISLs is running an incompatible firmware version and must be upgraded to Fabric OS v6.2.0. Base switch issues Symptom current base switch and then create a new one. Fabric OS Troubleshooting and Diagnostics Guide 71 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 86
-based blade into a non-default switch. To determine if you have a Condor-based blade, refer to the Fabric OS Administrator's Guide. Validation of switch configuration changes is not supported on this Verify the ports and try again. 72 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 87
fails. A non-GE blade is within the slot supports Virtual Fabrics has a maximum number of logical switches that may be supported problem. Once the configuration discrepancy has been fixed, you may use slotPowerOff followed by slotPowerOn to recover. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 88
blade compatibility Gathering additional information For Virtual Fabric-related issues, use the following guidelines to gather additional data for your switch support provider: • Perform the supportSave command. • If not sure about the problem OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 89
recommended action Use the trunkDebug port1 port2 command to troubleshoot the problem, as shown in the following procedure. 1. Connect to support trunking. • A trunking license is required. • Trunking is not supported in switch interoperability mode. Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 90
ports are not the same speed or they are not set to an invalid speed. Manually set port speeds to a speed supported on both sides of the trunk. • The ports are not set to the same issuing the commands portDisable and portEnable. 76 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 91
point for you to troubleshoot your zoning problem. 1. Verify that you have a zone problem. 2. Determine the nature merge problem 1. Enter the switchShow command to validate that the segmentation is because of a zone issue. 2. Review " OS Troubleshooting and Diagnostics Guide 77 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 92
together by ISLs and do not communicate to each other. Each switch appears as a separate fabric when you use the fabricShow command. 78 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 93
Use to create a zone alias. Use to delete a zone alias. Use to create a zone configuration. Displays zoning configuration. Disables the active (effective) configuration Fabric OS Troubleshooting and Diagnostics Guide 79 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 94
; then, log in to the switches as admin. 2. Enter the switchDisable command on all problem switches. 3. Enter the cfgDisable command on each switch. 4. Enter the cfgClear command on each database. The fabrics will then merge. 80 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 95
the fabric zone configuration for the segmented switch (see Table 14 on page 79 for specific commands. If the zoneset members between two switches are not listed in the same order zone members in alphabetical order, if possible. Fabric OS Troubleshooting and Diagnostics Guide 81 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 96
problems Enter the fcPing command (refer to the Fabric OS Command Reference for more information on this command), which checks the zoning configuration for the two ports specified by: • Generates an ELS (Extended Link Service do not support the Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 97
Reference. Checking for zoning problems 1. Enter the cfgActvShow command to the cfgSave command. 2. Confirm that the specific edge devices that must communicate with each to determine whether the name server is aware that these devices are Troubleshooting and Diagnostics Guide 83 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 98
Gathering additional information Collect the data from a supportSave -n command. Then collect the data from the cfgTransShow command. For the port having the problem, collect the data from the filterPortShow command. 84 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 99
power-on self test (POST). • Automatically on an individual blade whenever it is installed into a director chassis. • Manually and status information to the session screen, where you can review it or capture its data. If you are using a Troubleshooting and Diagnostics Guide 85 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 100
10 Power-on self test To save a set of files that customer support technicians can use to further diagnose the switch condition, enter the supportSave command. The command prompts for an FTP server, packages the following files, and sends them to the specified server: • The output of the - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 101
Power-on self test 10 eth0: Link status change: Link Up. 100 Mbps Full to bypass POST or after POST completes, various system services are started and the boot process displays additional console status and progress messages. Fabric OS Troubleshooting and Diagnostics Guide 87 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 102
PM Power supplies monitorDOWN Temperatures monitor HEALTHY Fans monitor DOWN WWN servers monitor HEALTHY Standby CP monitor HEALTHY Blades monitor HEALTHY Core Blades monitorHEALTHY Flash monitor HEALTHY Marginal ports monitorHEALTHY 88 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 103
. Displays logical switch attributes, including the fabric ID (FID) associated with the logical switch and the switch role (default switch or base switch). Fabric OS Troubleshooting and Diagnostics Guide 89 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 104
be fixed, negotiated, or auto-negotiated. The port status. Protocol support by GbE port. The details displayed for each switch differ on cumulative amount of uptime since the system was first powered on, and the load average over the past one Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 105
TABLE 15 Port type support Port type Supported in v6.2.0 Supported in v6.3.0 Supported in v6.4.0 Loopback test F_Ports connected to Brocade HBAs it is required that the firmware version on the HBA is v2.1.1 or later. Debugging spinFab OS Troubleshooting and Diagnostics Guide 91 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 106
(Offline test) on the failing port in-order to check whether the blade internal ports are having some problems. - The -lb_mode 1 operand verifies that the SFP is working within normal , pass 2, Pt0/17(7) Ch0/7 Receive Error/Timeout 92 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 107
command portDisable portnumber or portDisable slotnumber/portnumber. • To disable a port that is persistently enabled, enter the command portCfgPersistentDisable portnumber or portCfgPersistentDisable slotnumber/portnumber. Fabric OS Troubleshooting and Diagnostics Guide 93 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 108
to help diagnose if your switch is experiencing port problems. Viewing the status of a port 1. Connect to the switch and log in as admin. 2. Enter the portShow [slot/] port command, specifying the number that corresponds to the port you are troubleshooting. In this example, the status of port 10 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 109
FL_Port opened tenancies stopped due to starvation number of times FL has the tenancy number of times NL has the tenancy zero tenancy Fabric OS Troubleshooting and Diagnostics Guide 95 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 110
Frames received Encoding errors inside frames Frames with CRC errors CRC errors that occur on frames with good end-of-frame delimiters. 96 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 111
sensors depends on the switch type. For detailed specifications on these components, refer to the switch hardware reference manual. The specific output from the status commands varies depending on the switch type. Checking the temperature, fan, and power supply 1. Log in to the switch as user - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 112
manual for power supply status line. Checking temperature status 1. Connect to the switch and log in as admin. 2. Enter the tempShow command: switch:admin> tempshow Sensor State Centigrade Fahrenheit ID 1 Ok 28 82 2 Ok 16 60 98 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 113
to the hardware reference manual for your switch to to be saved across power cycles and reboots. each of the two CP blades. Since all RASlog messages will Addr: 192.168.38.2050 Since RASlog supports Virtual Fabrics and logical switches, the FID Troubleshooting and Diagnostics Guide 99 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 114
log entries overwrite the oldest log entries. The port log is not persistent and is lost over power-cycles and reboots. If the port log is disabled, an error message displays. NOTE Port log ,02e401c2,00000c1e,0000001c,00000000 100 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 115
portLogDump command output (trace) is a powerful tool that is used to troubleshoot fabric issues. The portLogDump output provides by the frame and is determined by the higher-level service or FC_4 upper level protocol. There are many different Troubleshooting and Diagnostics Guide 101 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 116
logging daemon (syslogd) is an IP-based service for logging system messages made available by optionally, setting the facility level. Configuring the host Fabric OS supports a subset of UNIX-style message severities that default to the Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 117
switch so that diagnostic information is transferred automatically to a remote server. If a problem occurs, you can then provide your customer support representative with the most detailed information possible. To ensure the best service, you should set up for automatic transfer as part of standard - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 118
problem. After the setup is complete, you can run the supportSave -c command to save RASLog, TRACE, supportShow, core file, FFDC data and other diagnostic support information to the server without specifying server y, no, n): [no] y 104 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 119
Switch Type and Blade ID Appendix A The switchType is a displayed field listed when you run the switchShow command. When you are gathering information to give to your switch support provider, you may switch 4 Gb 32-port switch Fabric OS Troubleshooting and Diagnostics Guide 105 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 120
n/a n/a n/a n/a Condor Condor Description 2 Gb 16-FC ports blade 4 Gb 16-FC ports blade 4 Gb 32-FC ports blade 8 Gb 16-FC ports blade 16 FC-ports with 2 GbE ports router blade 8 FC ports and 8 GbE copper RJ-45 ports iSCSI blade 106 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 121
24-FC ports on an application blade that provides Converged Enhanced Ethernet to bridge a Fibre Channel and Ethernet SAN. 24-FC port with 10 1-GbE and two 10-GbE ports Fibre Channel routing and FCIP blade 8 Gb 64-FC ports blade Fabric OS Troubleshooting and Diagnostics Guide 107 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 122
A Switch Type and Blade ID 108 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 123
b3:84 Port Index: 6 Share Area: No Device Shared in Other AD: No Redirect: No The Local Name Server has 1 entry } 1. Separate the triplets: 61 06 00 2. Convert each hexadecimal value to a decimal representation 97,06,00 Fabric OS Troubleshooting and Diagnostics Guide 109 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 124
Decimal 171 172 173 174 175 176 177 178 179 180 Hex ab ac ad ae af b0 b1 b2 b3 b4 110 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 125
Hex f1 f2 f3 f4 f5 f6 f7 f8 f9 fa Decimal 251 252 253 254 255 Hex fb fc fd fe ff Fabric OS Troubleshooting and Diagnostics Guide 111 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 126
B Hexadecimal 112 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 127
blade errors AP blade type 24 is inserted, 73 faulty, 3, 54 stuck in the 'LOADING' state, 3, 54 browser troubleshooting certificates, 65 C certificates corrupt, 65 invalid, 65 not installed, 65 troubleshooting, 65 command output, 7 common problem switch support provider POST, 86 power supply status, - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 128
errors, 57 AP blade type 24 is inserted, 73 blade application firmware failed, 61 blade application firrmware failed, 61 blade errors, 61 blade is faulty, 54 blade is stuck in the connection, 26 loop initialization failure, 29 114 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01 - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 129
Name Server, problems, 4 PLOGI, 24 POD enabling ports, 93 point-to-point initialization failure, 29 port, 93 bypassed, 30 disabled, 30 enabling, 93 in wrong mode, 30 initialization, 23 loopback, 30 Port cannot be moved, 4 port information, viewing, 94 Fabric OS Troubleshooting and Diagnostics Guide - Dell PowerEdge M610 | Fabric OS Troubleshooting and Diagnostics Guide - Page 130
and power supply, 97 temperature, status of, 98 test a port, 36 a switch, 36 TI zone problem, 77 time is not in-sync, 11 troubleshooting certificates, content mismatch, 5 resolving conflicts, 79 troubleshooting, 79 type mismatch, 5 116 Fabric OS Troubleshooting and Diagnostics Guide 53-1001769-01
53-1001769-01
30 March 2010
®
Fabric OS
Troubleshooting and Diagnostics Guide
Supporting Fabric OS v6.4.0