Home » Cisco Manuals » Bridges & Routers » Cisco CRS-1 » Manual Viewer

Cisco CRS-1 Troubleshooting Guide - Page 83

Troubleshooting Booting, Booting Tips

Add to My Manuals
Save this manual to your list of manuals

Page 83 highlights

2 C H A P T E R Troubleshooting Booting This chapter describes techniques that you can use to troubleshoot a router running Cisco IOS XR software. It includes the following sections: • Booting Tips, page 2-73 • Verifying Successful Bootup, page 2-74 • Verifying and Troubleshooting the Console Connection, page 2-74 • Verifying and Troubleshooting Route Processor and Shelf Controller Cards, page 2-75 Booting Tips The following booting sequence should be followed. See Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for detailed information on the booting sequence for routers running Cisco IOS XR software. 1. Connect to the console port. 2. Boot the route processor (RP) of the designated shelf controller (DSC). For Cisco CRS-1 Multishelf Systems, when the RP of the DSC boots up, if the RP or shelf controller (SC) on other chassis are configured to autoboot (for example, they are in a loop sending minimum boot image [MBI] requests), they boot automatically. If they have not been configured to autoboot, enter reset at the ROM Monitor (ROMMON) prompt. 3. Verify that the value of the configuration register (confreg) is 0x102. Enter confreg at the ROMMON prompt to display the configuration register value. If the configuration register value is not set to 0x102, see Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for information on changing the configuration register value. 4. It takes approximately 20 minutes for the cards on all chassis to be displayed in the Cisco IOS XR software show commands. Managed nodes (service processor [SP], line card [LC], and distributed route processor [DRP]) are booted by shelfmgr, mbimgr, and instdir automatically when ROMMON is set to autoboot. Use the following chassis numbering guidelines when numbering the chassis in a Cisco CRS-1 Multishelf System: Note It is recommended that the line card chassis containing the active DSC be numbered 0, the second line card chassis be numbered 1, and the fabric card chassis be numbered F0. See Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router for more information on recommended chassis numbering. OL-21483-02 Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router 2-73

Section	Page
Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router	1
Contents	3
Preface	9
General Troubleshooting	11
Troubleshooting Techniques and Approaches	11
Documenting the Network	12
Verifying and Troubleshooting CLI Access	12
General CLI Access Information	12
User Access Privileges	13
CLI Access Through a Console Port	13
CLI Access Through a Terminal Server	13
CLI Access Through the Management Ethernet Interface	14
Basic Cisco IOS XR Verification and Troubleshooting Tools	17
man Command	17
describe Command	19
show platform Command	20
top Command	20
show context Command	21
show users Command	23
Verifying the System	23
Validating and Troubleshooting Cisco IOS XR Software Package Installation	37
Verifying the Software Version	37
Validating the Installation	40
install verify Command	40
show install active Command	42
show install committed Command	43
Validating and Troubleshooting Cisco IOS XR Software Configuration	45
Local and Global Configurations	45
Collecting Configuration Information	47
Verifying the Running Configuration	48
Using the show configuration failed Command	57
Startup Failed Configuration	57
Commit Configuration Failed	58
ASIC Errors	59
Trace Commands	66
Packets	67
Logging Archive for Harddisk	70
SNMP Polling Awareness of SystemOwner, LR Owner, MIB Location	70
Error File Locations and Data Collection Scripts	70
Error File Locations	71
harddisk:	71
Default disk location	72
Sysmgr Collection Scripts	72
Wdsysmon Collection Scripts	73
Shutdown Collection Scripts	76
ASIC error Collection Scripts	76
Monitoring	76
monitor interface Command	77
monitor controller Command	78
monitor processes Command	79
monitor threads Command	80
Gathering Information for Technical Support	80
Troubleshooting Booting	83
Booting Tips	83
Verifying Successful Bootup	84
Verifying and Troubleshooting the Console Connection	84
Verifying and Troubleshooting Route Processor and Shelf Controller Cards	85
Troubleshooting RP and SC Cards Not Booting	85
Troubleshooting RP and SC Cards Resetting While Booting	88
Troubleshooting Blocked FCC Shelf Controller or LCC Route Process Minimum Boot Image Requests	89
Troubleshooting Forwarding	91
Troubleshooting IPv4 CEF Information	91
Troubleshooting Adjacency Information	95
Troubleshooting Transient Traffic Drop in Forwarding	98
Troubleshooting Control Plane Information	100
Troubleshooting the Interface Manager	105
Interface Manager Control Process	106
Troubleshooting the Trace Logs for the Interface Manager	110
Troubleshooting the Client for the Interface Manager	111
Troubleshooting the Rules for the Interface Manager	111
Troubleshooting the Control Chain and Interface Information	113
Troubleshooting the Registrations for the Interface Manager	115
Troubleshooting the Interface Manager Distributor	116
Interface Manager Distributor Overview	116
Troubleshooting the Trace Logs for the Interface Manager Distributor	117
Troubleshooting the Clients for the Interface Manager Distributor	117
Troubleshooting the Interface Information for the Interface Manager Distributor	118
Troubleshooting the Global Registrations for the Interface Manager Distributor	119
Troubleshooting the Rules for the Interface Manager Distributor	120
Troubleshooting Router Switch Fabric	121
CRS-1 Switch Fabric Overview	121
Understanding the Flags Field	123
Using the Online Diagnostics Tools	125
Verifying and Troubleshooting the Fabric Plane State	125
Verifying and Troubleshooting Up Fabric Planes	131
Troubleshooting Down Fabric Planes	136
Guidelines for Maintenance of Fabric Links	140
Troubleshooting Interfaces	141
Verifying and Troubleshooting Configured Interfaces	141
Verifying and Troubleshooting Pluggable Optical Interfaces	149
Troubleshooting Common Issues with Bundle Interfaces	154
Troubleshooting the Control Plane Ethernet Network	157
Cisco CRS-1 Control Plane Ethernet Network Overview	157
Using the Online Diagnostics Tools	158
Troubleshooting Booting the System Control Plane Ethernet Network	159
Examples	161
Troubleshooting the Multishelf System Router Topology	163
Examples	166
Troubleshooting the Catalyst Switch	168
Restrictions	168
Troubleshooting the CRS-1, 4-slot, 8-slot, or 16-slot System Router Topology	169
Examples	172
Collecting System Information	175
Capturing Logs	175
Using ping and traceroute	176
Using Debug Commands	176
Using Diagnostic Commands	176
Online Diagnostics	176
Transient Condition when Standby RP Becomes Active	178
Offline Diagnostics-FDIAG RUNNING State	179
Additional Reference for Diagnostic Commands	179
Commands Used to Display Process and Thread Details	179
Process Monitoring and Troubleshooting	181
System Manager	182
Watchdog System Monitor	182
Deadlock detections	182
Hang detection	182
Core Dumps	183
follow Command	183
show processes Commands	185
show processes boot Command	185
show processes startup Command	186
show processes failover Command	187
show processes blocked Command	188
Example:	189
Redundancy and Process Restartability	189
Process States	190
Synchronous Message Passing	191
Blocked Processes and Process States	191
Process Monitoring	192
Process Monitoring Commands	193
Monitoring CPU Usage and Using Syslog Messages	193
Troubleshooting High CPU Utilization and Process Timeouts	195
General Guidelines for Troubleshooting CPU Utilization Problems	195
Using show process and top processes Commands	196
Troubleshooting a Process Block	198
Troubleshooting a Process Crash on Line Cards	202
Troubleshooting a Memory Leak	203
Troubleshooting a Hardware Failure	204
Troubleshooting SNMP Timeouts	204
Troubleshooting Communication Among Multiple Processes	204
Troubleshooting a Process Restart	205
Troubleshooting Memory	207
Watchdog System Monitor	207
Memory Monitoring	207
Configuring and Displaying Memory Thresholds	208
Setting Timeout for Persistent CPU Hogs	211
Memory Usage Analyzer	212
Troubleshooting Global Memory	212
Troubleshooting Process Memory	213
Identifying Process Memory Problems	213
Resolving Process Memory Problems	217
Troubleshooting Upgrading and Downgrading Software	219
Validating and Troubleshooting ROM Monitor Software Installation	219
Verifying and Troubleshooting the ROM Monitor Version	220
Troubleshooting Upgrading and Downgrading ROM Monitor Software on Cisco CRS-1 Routers	223
Troubleshooting Upgrading and Downgrading Cisco IOS XR Software Packages	224
Troubleshooting the Statistics Infrastructure	225
Debugging Statistics Infrastructure	225
debug statsd api Commands	225
debug statsd manager Commands	226
debug statsd server errors Command	226
Trace Commands for the Statistics Infrastructure	227
show statsd manager trace Command	227
show statsd server trace Command	228
show stats lib trace	228
Show Commands for the Statistics Infrastructure	228
show statsd manager info Command	229
show statsd collectors Command	230
show statsd registrations Command	232
show statsd requests Command	232
Diagnosing Problems with Statistics Values	233
Errors When Retrieving Data from the Statistics Manager EDM	234
Timeouts and Delays When Retrieving Data	234
Displaying Incorrect Rates for the show interfaces Command	235
Displaying Incorrect Data for the show Commands	236
Multiprotocol Label Switching Checklist	239
Multiprotocol Label Switching Recommendations	239
Troubleshooting L3VPN MPLS Traffic Loss	240
Troubleshooting Load Balancing	241
About Load Balancing with Cisco Express Forwarding	241
Load Balancing Function	241
Source Information for Load Balancing	241
Layer 2 Load Balancing	242
Terminology	243
Troubleshooting Layer 3 or Layer 4 Load Balancing	243
Verifying the Routing Table Entries for Parallel Links	244
Configuring Layer 4 Load Balancing	246
Verifying the CEF Database and Measuring Flows	246
Troubleshooting Layer 2 Load Balancing	247
Verifying the Bundle Status, IGP Route, and CEF Database	247
Configuring Layer 4 Load Balancing	248
Viewing the Expected Paths and Measuring the Flows	248
Configuration Examples for Troubleshooting Load Balancing	249
Troubleshooting Layer 3 or Layer 4 Load Balancing Example	250
Verifying Parallel Links in the Routing Table	250
Verifying IPv4/IPv6 Unicast Load Balancing	253
Troubleshooting Layer 2 Load Balancing Example	256
Verifying the Bundle Status	256
Checking the IGP Routing Table and CEF Database	257
Verifying Bundle Load Balancing	257

Match case Limit results 1 per page

CHAPTER

2-73

Cisco IOS XR Troubleshooting Guide for the Cisco CRS-1 Router

OL-21483-02

Troubleshooting Booting

This chapter describes techniques that you can use to troubleshoot a router running Cisco IOS XR

software. It includes the following sections:

•

Booting Tips, page 2-73

•

Verifying Successful Bootup, page 2-74

•

Verifying and Troubleshooting the Console Connection, page 2-74

•

Verifying and Troubleshooting Route Processor and Shelf Controller Cards, page 2-75

Booting Tips

The following booting sequence should be followed. See

Cisco IOS XR Getting Started Guide for the

Cisco CRS-1 Router

for detailed information on the booting sequence for routers running Cisco IOS XR

software.

Connect to the console port.

Boot the route processor (RP) of the designated shelf controller (DSC).

For Cisco CRS-1 Multishelf Systems, when the RP of the DSC boots up, if the RP or shelf controller

(SC) on other chassis are configured to autoboot (for example, they are in a loop sending minimum

boot image [MBI] requests), they boot automatically. If they have not been configured to autoboot,

enter

reset

at the ROM Monitor (ROMMON) prompt.

Verify that the value of the configuration register (confreg) is 0x102. Enter

confreg

at the

ROMMON prompt to display the configuration register value. If the configuration register value is

not set to 0x102, see

Cisco IOS XR Getting Started Guide for the Cisco CRS-1 Router

for

information on changing the configuration register value.

It takes approximately 20 minutes for the cards on all chassis to be displayed in the Cisco IOS XR

software

show

commands. Managed nodes (service processor [SP], line card [LC], and distributed

route processor [DRP]) are booted by shelfmgr, mbimgr, and instdir automatically when ROMMON

is set to autoboot.

Use the following chassis numbering guidelines when numbering the chassis in a

Cisco CRS-1 Multishelf System:

Note

It is recommended that the

line card chassis containing the active DSC be numbered 0, the second line card

chassis be numbered 1, and the fabric card chassis be numbered F0

. See

Cisco IOS XR Getting Started

Guide for the Cisco CRS-1 Router

for more information on recommended chassis numbering.