Home » Cisco Manuals » Hubs & Switches » Cisco N5K-C5010P-BF » Manual Viewer

Cisco N5K-C5010P-BF Troubleshooting Guide - Page 125

Troubleshooting Fibre Channel MAC issues, show hardware

UPC - 882658212208

View all Cisco N5K-C5010P-BF manuals

Add to My Manuals
Save this manual to your list of manuals

Page 125 highlights

Chapter 5 Troubleshooting SAN Switching Issues Registers and Counters Send document comments to [email protected]. Troubleshooting Fibre Channel MAC issues The show hardware command is very useful when troubleshooting FC physical layer issues. Show hardware internal fc-mac port statistics The output of the command contains the following useful information: • Physical layer information FCP_CNTR_MAC_RX_LOSS_OF_SYNC - Loss of Sync received counter • Performance information FCP_CNTR_MAC_CREDIT_IG_XG_MUX_SEND_RRDY_REQ - Receiver Ready's Sent FCP_CNTR_MAC_CREDIT_EG_DEC_RRDY - Receiver Ready's Received • Class 3 normal traffic counters FCP_CNTR_MAC_DATA_RX_CLASS3_FRAMES FCP_CNTR_MAC_DATA_RX_CLASSF_FRAMES FCP_CNTR_MAC_DATA_RX_CLASS3_WORDS FCP_CNTR_MAC_DATA_RX_CLASSF_WORDS FCP_CNTR_MAC_DATA_TX_CLASS3_FRAMES FCP_CNTR_MAC_DATA_TX_CLASSF_FRAMES FCP_CNTR_MAC_DATA_TX_CLASS3_WORDS FCP_CNTR_MAC_DATA_TX_CLASSF_WORDS • Fibre Channel primitive sequences FCP_CNTR_LINK_RESET_IN - Link Resets Received FCP_CNTR_OLS_OUT- Offline Sequences Sent FCP_CNTR_NOS_OUT - Not Operational Sequence Sent FCP_CNTR_LRR_OUT - Link Reset Responses Sent FCP_CNTR_LINK_FAILURE Example: switch# show hardware internal fc-mac 2 port 1 statistics ADDRESS STAT 0x0000003c FCP_CNTR_MAC_RX_LOSS_OF_SYNC 0x0000003d FCP_CNTR_MAC_RX_BAD_WORDS_FROM_DECODER 0x00000042 FCP_CNTR_MAC_CREDIT_IG_XG_MUX_SEND_RRDY_REQ 0x00000043 FCP_CNTR_MAC_CREDIT_EG_DEC_RRDY 0x00000061 FCP_CNTR_MAC_DATA_RX_CLASS3_FRAMES 0x00000062 FCP_CNTR_MAC_DATA_RX_CLASSF_FRAMES 0x00000069 FCP_CNTR_MAC_DATA_RX_CLASS3_WORDS 0x0000006a FCP_CNTR_MAC_DATA_RX_CLASSF_WORDS 0x00000065 FCP_CNTR_MAC_DATA_TX_CLASS3_FRAMES 0x00000066 FCP_CNTR_MAC_DATA_TX_CLASSF_FRAMES 0x0000006d FCP_CNTR_MAC_DATA_TX_CLASS3_WORDS 0x0000006e FCP_CNTR_MAC_DATA_TX_CLASSF_WORDS 0xffffffff FCP_CNTR_LINK_RESET_IN 0xffffffff FCP_CNTR_OLS_OUT 0xffffffff FCP_CNTR_NOS_OUT 0xffffffff FCP_CNTR_LRR_OUT 0xffffffff FCP_CNTR_LINK_FAILURE COUNT 0x5 0xec 0x5ec 0xc41 0x5d2 0x1a 0x140b14 0xdcc 0xc24 0x1d 0x4b9538 0xabc 0x2 0x5 0x2 0x7 0x2 OL-25300-01 Cisco Nexus 5000 Series Troubleshooting Guide 5-53

Section	Page
Cisco Nexus 5000 Series Troubleshooting Guide	1
Contents	3
Preface	5
Audience	5
Organization	5
Document Conventions	6
Related Documentation	7
Troubleshooting Overview	9
Troubleshooting Basics	9
Troubleshooting a Switch Crash	9
Best Practices	11
Common Terms	11
Fabric Manager Tools and CLI Commands	12
NX-OS Tips	12
Displaying what is required from the configuration	12
Displaying within Config Mode	12
Pipe command	12
Using the pipe command to only display required keyword	13
Copy command	13
Redirecting output	13
Redirecting output of the show tech-support details command	13
NX-OS command listing	14
Narrowing scope of keywords	14
Logging	14
Ethanalyzer and SPAN	15
Ethanalyzer	15
SPAN	17
Source Ports	18
SPAN Destinations	18
Characteristics of Destination Ports	18
Monitor Caveats	19
SPAN Configuration	19
Verifying the SPAN Session	19
Suspending the SPAN Session	20
Debugging	20
Command-Line Debugging	20
Debug Logging	20
Debugs to the Direct Telnet Window	21
Cisco Discover Protocol	21
Failover	22
FCoE Traffic	22
Non-FCoE traffic	22
LAN Traffic	23
Troubleshooting FCoE Issues	25
Data Center Bridging	25
VFC (FCoE) interface not online	25
General troubleshooting	25
Nexus 5548 Troubleshooting	30
FIP	31
VFC down due to FIP failure	31
VFC down due to FIP solicitation failure	31
VFC down because VLAN response not received by CNA	32
VFC down because no active STP port-state on the bound Ethernet interface	32
VFC down due to FIP keepalive misses	32
CNA	33
Best practice topology for CNA	33
Troubleshooting with Host tools	34
CNA not recognized by Host OS	34
PFC	35
Standard pause frames	35
PFC not negotiated with FCOE-capable adapters (CNA)	35
Switch Interface connected to CNA receives constant pause frames (PFC)	36
Check if switch is sending pause frames or getting paused	37
Switch ports err-disabled due to pause rate-limit	38
How to enable link pause (flow control) on switch that connects DCBX capable devices	39
How to clear PFC counters	39
Registers and Counters	39
Interface level errors	39
Packet byte counts	40
Verification of SNMP readouts	41
Traffic rates	41
Troubleshooting Layer 2 Switching Issues	43
MAC Address Table	43
Data traffic is flooding	43
MAC address not learned	44
Traffic flooding in a VPC setup	45
Spanning Tree Protocol	45
HIFs go down with the BPDUGuard errDisable message	45
FWM-2-STM_LOOP_DETECT detected on switch, dynamic learning disabled	45
Port stuck in STP blocking state with BLK*(Type_Inc)	46
Port stuck in STP blocking state with BLK*(PVID_Inc)	46
Port stuck in STP blocking state with BLK*(Loop_Inc)	46
Multicast	47
Source MAC addresses of IGMP joins are learned	47
Multicast data traffic not received by host	47
Multicast data traffic not received when host is registered for group	47
Multicast traffic is being flooded in a VPC setup	48
VLANs	48
Nexus 5000 does not have the same VLANs as switch running VTP server	48
VLAN cannot be created	48
Interface VLAN is down	49
Configuring interface to access port does not allow VLAN <###> to go through	50
Cannot create VLAN	50
Cannot create SVI	51
Cannot create private VLAN (PVLAN)	51
Registers and Counters	52
Identifying drops	52
Expected/Logical drops	52
Queue is full	53
MTU violation	54
Handling CRC errors	54
MAC Statistics	54
Troubleshooting QoS Issues	59
Policy Maps	59
Improper Configurations	60
Cannot pass frame size larger than 2300 bytes through switch	60
MTU for “class-default” value is 1500 when jumbo MTU configured	61
Traffic not queued or prioritized correctly on Nexus 2148, Nexus 2232, and Nexus 2248	61
PFC	66
Link pause (flow control) not enabled on back to back Nexus 5000 switch links	66
Cannot enable “pause no-drop” on more than one ethernet class	66
Changing no-drop configuration causes VPC peer-link to go down and FEX to go offline	67
Pause enabled on all cos values when no-drop enabled on class-ip-multicast	67
No drop class not created on N2K-C2148T/N2K-C2248TP-1GE based FEX with default QoS configuration	68
How to enable link pause (flow control) on Nexus 5000 interface	68
Registers and Counters	68
Nexus 5000 10G PFC	69
Nexus 5000 1G storm control	69
Nexus 5000 10G storm control	69
Nexus 5000 storm control counter	69
afm-related CLI commands and tools	69
FEX qosctrl debug commands	70
N2K-C2148T FEX counters	70
Nexus 5000 multicast-optimization	71
Nexus 5000 FCoE classification	71
Nexus 5000 MTU programming	71
Nexus 5000 interrupt	71
Untagged COS	72
Buffer usage and packet drop debugging on N2K-C2232P FEX	72
Troubleshooting SAN Switching Issues	73
Overview	73
General SAN troubleshooting steps	74
NPV	74
NP Uplink ports on NPV edge switch are stuck in initializing state	74
Server interface does not come up and “NPV upstream port not available” message appears	75
Uneven load balancing on the NPV NP ports	75
Server on downstream NPV edge switch does not login to the fabric	76
Locating exact port that server is physically attached to	77
VSANs stuck in initializing state after configuring the 4.2(1)N1 F_Port trunking feature	78
Zoning	79
Cannot activate zoneset and cannot configure zoning in enhanced zoning mode	79
Host cannot communicate with storage	80
Zone merge failure when two switches connect using E or TE port	82
Zone set activation failure	83
Full zone database synchronization failure across two switches	84
Mismatched default zone policy in switches in VSAN causes unexpected results when accessing storage	84
SAN Port Channels	85
Fibre channel port is down when trying to connect switches via SAN Port Channel	85
Newly added Fibre Channel interface does not come online in a SAN Port Channel	86
Cannot configure trunking	86
VSAN traffic does not traverse trunk	86
xE port is isolated in a specific VSAN under interface of SAN Port Channel	87
Cannot create a san-port-channel interface	87
FC Services	88
Overview	88
Fibre channel port remains in initializing state	89
Specific VSAN traffic is not being routed through SAN fabric	90
Fibre channel port is suspended due to too many invalid FLOGIs	95
Having stale FCNS entries for Fibre Channel nodes	98
Interface is isolated because of FC Domain ID overlap	100
Cisco Fabric Services	103
Overview	103
Verifying CFS using CLI	103
Merge failure troubleshooting	106
Recovering from a Merge Failure with the CLI	107
Lock failure troubleshooting	108
Resolving lock failure issues using the CLI	109
System state inconsistent and locks being held	110
Clearing locks using the CLI	110
Distribution status verification	110
Verifying distribution using the CLI	110
CFS regions troubleshooting	110
Distribution failure	111
Regions for conditional service	112
Changing regions	112
VSANs	112
Overview	112
VSAN Troubleshooting Activities	113
Common troubleshooting tools in Fabric Manager	113
Common troubleshooting CLI commands	113
Checklist	113
Nexus 5000 trunk port does not connect to upstream SAN switch	114
Nexus 5000 E port (non-trunking) does not connect to upstream SAN switch	116
Communication problem between host and storage devices	118
VSAN is down between switches	118
Registers and Counters	120
Identifying physical layer issues	120
Displaying FcoE bound Ethernet interface counters	121
Understanding Fibre Channel interface counters	124
Troubleshooting Fibre Channel MAC issues	125
Troubleshooting Fibre Channel forwarding issues	126
Troubleshooting Security Issues	129
Roles	129
Role assignment fails when user logs in	129
Rules for Role's permit/deny action do not work correctly	130
Role's interface or VLAN policy does not appear to work correctly	131
Assigning multiple roles to single user does not seem to work correctly	131
Change to role configuration does not get applied	131
CLI rejects feature-group removal	132
AAA	132
User cannot login through TACACS+ or RADIUS authentication	132
Unable to decode content of packets with Wireshark	133
Role assignment fails when user logs in	133
No command accounting logs on ACS server when TACACS+ accounting enabled	134
PAP authentication does not work for RADIUS	134
Authentication fallback method appears inoperable	135
Troubleshooting System Management Issues	137
SNMP	137
SNMP memory usage continuously increasing	137
SNMP not responding	138
SNMP not responding and show snmp command reports SNMP has timed out	138
Not able to perform SNMP SET operation	138
SNMP on BRIDGE-MIB	139
Logging	139
System is not responsive	139
Syslog server not getting messages from DUT	139
Traps	140
Traps not received	140
DNS	140
DNS resolution not working correctly	140
Specified domain not removed from domain-list	141
Troubleshooting Virtual Port Channel Issues	143
Improper Configurations	143
vPC fails to start	143
Unable to configure vPC	143
vPC in blocking state	144
vPC domain ids	144
Connectivity issues	145
Peer-link issues	147
vPC Consistency parameter issues	149
Troubleshooting Config-Sync Issues	151
Commit Failure	151
Command Parsing Failed	151
Verify failed	152
Commands that failed commit	152
Another session in progress	152
Import Failure	153
Failed to collect running-config	154
Command does not exist in global-db	154
Mutual exclusion check failed on peer	155
Merge Failure	155
First time merge failure	156
Merge after peers that were in sync previously	156
Merge after reload	156
Switch-profile Deletion Failure	156
Application failure	157
Failure from dependent commands	157
Application does not respond	157
Verify Failure	158
Mutual exclusion check under local information	158
Mutual exclusion check under peer information	158
Rollback/ISSU in progress	158
Global-db modification in progress	159
Peer unable to accept lock request	159

Match case Limit results 1 per page

Send document comments to [email protected].

5-53

Cisco Nexus 5000 Series Troubleshooting Guide

OL-25300-01

Chapter 5

Troubleshooting SAN Switching Issues

Registers and Counters

Troubleshooting Fibre Channel MAC issues

The

show hardware

command is very useful when troubleshooting FC physical layer issues.

Show hardware internal fc-mac

<x>

port

<y>

statistics

The output of the command contains the following useful information:

•

Physical layer information

FCP_CNTR_MAC_RX_LOSS_OF_SYNC - Loss of Sync received counter

•

Performance information

FCP_CNTR_MAC_CREDIT_IG_XG_MUX_SEND_RRDY_REQ - Receiver Ready's Sent

FCP_CNTR_MAC_CREDIT_EG_DEC_RRDY - Receiver Ready's Received

•

Class 3 normal traffic counters

FCP_CNTR_MAC_DATA_RX_CLASS3_FRAMES

FCP_CNTR_MAC_DATA_RX_CLASSF_FRAMES

FCP_CNTR_MAC_DATA_RX_CLASS3_WORDS

FCP_CNTR_MAC_DATA_RX_CLASSF_WORDS

FCP_CNTR_MAC_DATA_TX_CLASS3_FRAMES

FCP_CNTR_MAC_DATA_TX_CLASSF_FRAMES

FCP_CNTR_MAC_DATA_TX_CLASS3_WORDS

FCP_CNTR_MAC_DATA_TX_CLASSF_WORDS

•

Fibre Channel primitive sequences

FCP_CNTR_LINK_RESET_IN - Link Resets Received

FCP_CNTR_OLS_OUT- Offline Sequences Sent

FCP_CNTR_NOS_OUT - Not Operational Sequence Sent

FCP_CNTR_LRR_OUT - Link Reset Responses Sent

FCP_CNTR_LINK_FAILURE

Example:

switch# show hardware internal fc-mac 2 port 1 statistics

ADDRESS

STAT

COUNT

__________ ________

__________________

0x0000003c FCP_CNTR_MAC_RX_LOSS_OF_SYNC

0x5

0x0000003d FCP_CNTR_MAC_RX_BAD_WORDS_FROM_DECODER

0xec

0x00000042 FCP_CNTR_MAC_CREDIT_IG_XG_MUX_SEND_RRDY_REQ

0x5ec

0x00000043 FCP_CNTR_MAC_CREDIT_EG_DEC_RRDY

0xc41

0x00000061 FCP_CNTR_MAC_DATA_RX_CLASS3_FRAMES

0x5d2

0x00000062 FCP_CNTR_MAC_DATA_RX_CLASSF_FRAMES

0x1a

0x00000069 FCP_CNTR_MAC_DATA_RX_CLASS3_WORDS

0x140b14

0x0000006a FCP_CNTR_MAC_DATA_RX_CLASSF_WORDS

0xdcc

0x00000065 FCP_CNTR_MAC_DATA_TX_CLASS3_FRAMES

0xc24

0x00000066 FCP_CNTR_MAC_DATA_TX_CLASSF_FRAMES

0x1d

0x0000006d FCP_CNTR_MAC_DATA_TX_CLASS3_WORDS

0x4b9538

0x0000006e FCP_CNTR_MAC_DATA_TX_CLASSF_WORDS

0xabc

0xffffffff FCP_CNTR_LINK_RESET_IN

0x2

0xffffffff FCP_CNTR_OLS_OUT

0x5

0xffffffff FCP_CNTR_NOS_OUT

0x2

0xffffffff FCP_CNTR_LRR_OUT

0x7

0xffffffff FCP_CNTR_LINK_FAILURE

0x2