Home » Cisco Manuals » Servers » Cisco MCS-7825-H3-IPC1 » Manual Viewer

Cisco MCS-7825-H3-IPC1 Service Guide - Page 138

Cisco CallManager Service Stops Responding

View all Cisco MCS-7825-H3-IPC1 manuals

Add to My Manuals
Save this manual to your list of manuals

Page 138 highlights

General Health and Troubleshooting Tips Chapter 3 Managing and Monitoring the Health of Cisco Unified Communications Manager Systems • Cisco RIS Data Collector PerfMonLog is disabled by default in Cisco Unified CM Release 5.x and enabled by default in Cisco Unified CM Release 6.0. Cisco CallManager Service Stops Responding When the Cisco CallManager service stops responding, the following message displays in the System Event log: The Cisco CallManager service terminated unexpectedly. It has done this 1 time. The following corrective action will be taken in 60000 ms. Restart the service. Other messages you may see in this situation: Timeout 3000 milliseconds waiting for Cisco CallManager service to connect. The Cisco Communications Manager failed to start due to the following error: The service did not respond to the start or control request in a timely fashion. At this time when devices such as the Cisco Unified IP Phones and gateways, unregister from the Cisco Unified Communications Manager, users receive delayed dial tone, and/or the Cisco Unified Communications Manager server freezes due to high CPU usage. For event log messages that are not included here, view the Cisco Unified Communications Manager Event Logs. Possible Cause The Cisco CallManager service can stop responding because the service does not have enough resources such as CPU or memory to function. Generally, the CPU utilization in the server is 100 percent at that time. Recommended Action Depending on what type of interruption you experience, you will need to gather different data that will help determine the root cause of the interruption. Use the following procedure if a lack of resources interruption occurs. Procedure Step 1 Step 2 Step 3 Step 4 Collect Cisco CallManager traces 15 minutes before and after the interruption. Collect SDL traces 15 minutes before and after the interruption. Collect perfmon traces if available. If the traces are not available, start collecting the perfmon traces and track memory and CPU usage for each process that is running on the server. These will help in the event of another lack of resources interruption. 3-38 Cisco Unified Communications Manager Managed Services Guide OL-22523-01

Section	Page
Cisco Unified Communications Manager Managed Services Guide, Release 8.5(1)	1
Preface	29
Purpose	29
Audience	29
Organization	30
Related Documentation	30
Conventions	31
Obtaining Documentation and Submitting a Service Request	32
Cisco Product Security Overview	32
Related Documentation	33
Overview	35
Cisco Unified Communications Manager	35
Supported Deployment Models	36
Managed Services	37
Cisco Unified Serviceability	38
Trace Tools	38
Troubleshooting Trace	39
Trace Collection	39
Cisco Unified Reporting	39
Cisco Unified Real-Time Monitoring Tool	40
Call Detail Records and Call Management Records	41
Call Detail Record Analysis and Reporting	41
Management Information Base	42
New and Changed Information	43
Cisco Unified Communications Manager, Release 8.5(1)	43
Audit Log Support for Cisco Unity Connection	44
Alarm Additions and Changes	45
Enhanced Reason Codes	46
New Perfmon Counters for Cisco SIP Normalization	48
SNMP MIBs	55
Supported Servers	55
Serviceability - Session Manager Edition (SME)	56
Default Settings for Alarm Configuration Settings	61
New Cisco Unity Connection Alerts	61
Logging On to CAR	62
Configuring the Trunk	63
Configuring Trunk Utilization Reports	63
Cisco Dialed Number Analyzer Server	65
Cisco Unified Communications Manager, Release 8.0(2)	66
Cisco Unified Communications Manager Server	66
Cisco Intercompany Media Engine Server	67
Cisco Unified Communications Manager, Release 8.0(1)	67
Cisco Unified Serviceability	68
New Alarm Additions and Changes	68
Obsolete Alarms	85
Cisco Unified Real-Time Monitoring Tool	89
Cisco Unified CDR Analysis and Reporting	93
New Cisco CAR DB Alarms	93
New CAR Object and Counters	93
Hunt/CTI Integration for CAR Reporting	94
CAR and CDRM Alarm Interface	94
System-Wide Call Tracking End-to-End Call Trace	94
Cisco Unified Call Detail Records	94
End-to-End Call Trace	95
Remote Destination to Number Mapping and CDRs	95
New CDR Fields to Support Call Control Discovery	95
New CDR Fields to Support External Call Control	95
New CDR Support for iSAC Codec	97
New CDR Fields for Hunt List Support	97
Cisco Unified Reporting	97
MIB Updates for 8.0(1)	98
Managing and Monitoring the Health of Cisco Unified Communications Manager Systems	101
Overview of Supported Interfaces	101
Critical Processes to Monitor	102
Available Supported MIBs	111
RTMT Monitoring of Cisco Unified CM System Health	112
RTMT Summary View	112
CPU Usage	113
% IOwait Monitoring	115
Virtual Memory	116
Disk Usage	117
Disk Name Mapping	118
Database Replication and Cisco Unified Communication Manager Nodes	120
Cisco Unified CM Process and CPU Usage	120
CodeYellow	121
RIS Data Collector PerfMonLog	123
Critical Service Status	124
Syslog Messages	125
RTMT Alerts as Syslog Messages and Traps	126
Recovery, Hardware Migration, and Backup/Restore	126
Backup/Restore	126
Platform Monitoring	127
Using SNMP MIBs	127
MIBs and MCS Types	128
Using Command Line Interface	128
Hardware Migration	131
Platform Security	131
Locked-down System	132
Cisco Security Agent Support	132
Security Patching and Updating	132
Role-Based Access Control	132
Software Configuration Management	132
General Install/Upgrade Procedures	133
Detecting Installed Release and Packages	133
Available Reports	133
RTMT Reports	133
Serviceability Reports	134
Cisco Unified Reporting	134
General Health and Troubleshooting Tips	135
Using of Onboard Agents	136
Call Detail Records and Call Maintenance Records	136
Perfmon Counters	136
Integration with Uninterruptible Power Supplies (UPS)	137
Native Hardware Out of Band Management (OOB)	137
Phone Registration Status	137
Historical Information Download	137
Cisco CallManager Service Stops Responding	138
Database Replication Fails Between the Publisher and the Subscriber	139
Database Replication Does Not Occur on Lost Node	141
Database Tables Out of Sync Do Not Trigger Alert	142
Reset Database Replication When Reverting to Prior Release	143
Useful Commands and Utilities	143
Related Documentation	144
Simple Network Management Protocol	147
Overview	147
SNMP Versioning	148
SNMP and Cisco Unified CM Basics	149
SNMP Basic Commands	149
SNMP Community Strings and Users	150
SNMP and Cisco MIBs	150
SNMP Traps and Informs	151
SNMP Trace Configuration	151
SNMP Tips	151
SNMP Troubleshooting	152
SNMP/R MIBs	154
Cisco Unified Real-Time Monitoring Tool Tracing, PerfMon Counters, and Alerts	155
Cisco Unified Real-Time Monitoring	155
Performance Monitoring in RTMT	156
PerfMon Alert Notifications	156
PerfMon Objects and Counters for Cisco Unified Communications Manager	159
Cisco Analog Access	159
Cisco Annunciator Device	159
Cisco CallManager	159
Cisco CallManager External Call Control	167
Cisco CallManager SAF	168
Cisco CallManager System Performance	169
Cisco CTIManager	171
Cisco Dual-Mode Mobility	171
Cisco Extension Mobility	173
Cisco Feature Control Policy	174
Cisco Gatekeeper	174
Cisco H.323	174
Cisco Hunt Lists	175
Cisco HW Conference Bridge Device	176
Cisco IP Manager Assistant	176
Cisco Lines	177
Cisco Locations	177
Cisco Media Streaming Application	178
Cisco Messaging Interface	181
Cisco MGCP BRI Device	182
Cisco MGCP FXO Device	183
Cisco MGCP FXS Device	183
Cisco MGCP Gateways	184
Cisco MGCP PRI Device	184
Cisco MGCP T1 CAS Device	185
Cisco Mobility Manager	186
Cisco Music On Hold (MOH) Device	187
Cisco MTP Device	188
Cisco Phones	188
Cisco Presence Feature	188
Cisco QSIG Feature	189
Cisco Signaling Performance	189
Cisco SIP	190
Cisco SIP Normalization	190
Cisco SIP Stack	197
Cisco SIP Station	205
Cisco SW Conf Bridge Device	206
Cisco TFTP Server	207
Cisco Transcode Device	210
Cisco Video Conference Bridge	211
Cisco Web Dialer	212
Cisco WSM Connector	212
PerfMon Objects and Counters for System	213
Cisco Tomcat Connector	213
Cisco Tomcat JVM	214
Cisco Tomcat Web Application	215
Database Change Notification Client	216
Database Change Notification Server	217
Database Change Notification Subscription	218
Database Local DSN	218
DB User Host Information Counters	218
Enterprise Replication DBSpace Monitors	218
Enterprise Replication Perfmon Counters	219
IP	219
Memory	220
Network Interface	221
Number of Replicates Created and State of Replication	222
Partition	223
Process	224
Processor	225
System	226
TCP	227
Thread	227
Cisco Intercompany Media Engine Performance Objects and Alerts	228
Cisco Intercompany Media Engine Server Objects	228
IME Configuration Manager	228
IME Server	228
IME Server System Performance	231
Cisco Intercompany Media Engine Server Alerts	232
Cisco Unified Communications Manager Server Objects	232
IME Client	233
IME Client Instance	234
Cisco Unified Communications Manager Server Alerts	234
Cisco Unified Serviceability Alarms and CiscoLog Messages	235
Cisco Unified Serviceability Alarms and CiscoLog Messages	236
CiscoLog Format	236
Log File and Syslog Outputs	237
Standard Syslog Server Implementations	238
Clock Synchronization	238
Multipart Messages	238
CiscoLog Message Format	239
Message Length Limit	240
SEQNUM Field	240
HOST Field	240
TIMESTAMP Field	242
HEADER Field	244
TAGS Field	248
MESSAGE Field	251
Internationalization	252
Versioning	252
Preconfigured System Alarm Notifications	253
AuthenticationFailed	253
CiscoDRFFailure	254
CoreDumpFileFound	255
CpuPegging	255
CriticalServiceDown	256
HardwareFailure	256
LogFileSearchStringFound	257
LogPartitionHighWaterMarkExceeded	257
LogPartitionLowWaterMarkExceeded	258
LowActivePartitionAvailableDiskSpace	259
LowAvailableVirtualMemory	259
LowInactivePartitionAvailableDiskSpace	260
LowSwapPartitionAvailableDiskSpace	260
ServerDown	261
SparePartitionHighWaterMarkExceeded	262
SparePartitionLowWaterMarkExceeded	263
SyslogSeverityMatchFound	264
SyslogStringMatchFound	264
SystemVersionMismatched	265
TotalProcessesAndThreadsExceededThreshold	265
Preconfigured CallManager Alarm Notifications	266
BeginThrottlingCallListBLFSubscriptions	267
CallProcessingNodeCpuPegging	267
CDRAgentSendFileFailed	268
CDRFileDeliveryFailed	269
CDRHighWaterMarkExceeded	269
CDRMaximumDiskSpaceExceeded	270
CodeYellow	270
DBChangeNotifyFailure	271
DBReplicationFailure	271
DDRBlockPrevention	272
DDRDown	273
ExcessiveVoiceQualityReports	273
IMEDistributedCacheInactive	274
IMEOverQuota	274
IMEQualityAlert	275
InsufficientFallbackIdentifiers	276
IMEServiceStatus	276
InvalidCredentials	277
LowCallManagerHeartbeatRate	278
LowTFTPServerHeartbeatRate	278
MaliciousCallTrace	279
MediaListExhausted	279
MgcpDChannelOutOfService	280
NumberOfRegisteredDevicesExceeded	280
NumberOfRegisteredGatewaysDecreased	281
NumberOfRegisteredGatewaysIncreased	281
NumberOfRegisteredMediaDevicesDecreased	282
NumberOfRegisteredMediaDevicesIncreased	283
NumberOfRegisteredPhonesDropped	283
RouteListExhausted	283
SDLLinkOutOfService	284
TCPSetupToIMEFailed	285
TLSConnectionToIMEFailed	285
Emergency-Level Alarms	286
BDINotStarted	286
CallDirectorCreationError	286
CiscoDirSyncStartFailure	287
ExcceptionInInitSDIConfiguration	287
FileWriteError	288
GlobalSPUtilsCreationError	288
HuntGroupControllerCreationError	288
HuntGroupCreationError	289
IPAddressResolveError	289
IPMANotStarted	290
LineStateSrvEngCreationError	290
LostConnectionToCM	290
NoCMEntriesInDB	291
NoFeatureLicense	291
OutOfMemory	292
ServiceNotInstalled	292
SyncDBCreationError	293
SysControllerCreationError	293
TapiLinesTableCreationError	293
TimerServicesCreationError	294
TestAlarmEmergency	294
WDNotStarted	295
Alert-Level Alarms	295
CertValidLessthanADay	295
CMIException	296
CMOverallInitTimeExceeded	296
ConfigThreadChangeNotifyServerInstanceFailed	297
ConfigThreadChangeNotifyServerSingleFailed	298
ConfigThreadChangeNotifyServerStartFailed	298
CiscoLicenseApproachingLimit	299
CiscoLicenseOverDraft	299
CMVersionMismatch	300
CreateThreadFailed	300
DBLException	301
InvalidCredentials	302
MemAllocFailed	303
NoDbConnectionAvailable	303
ParityConfigurationError	304
SerialPortOpeningError	304
SDIControlLayerFailed	305
SDLLinkOOS	306
SocketError	307
StopBitConfigurationError	307
TFTPServerListenSetSockOptFailed	308
TFTPServerListenBindFailed	308
TestAlarmAlert	309
TLSConnectionToIMEFailed	309
TVSServerListenBindFailed	310
TVSServerListenSetSockOptFailed	311
UnknownException	311
VMDNConfigurationError	312
Critical-Level Alarms	313
BChannelOOS	313
CallManagerFailure	314
CertExpiryCritical	315
CertValidfor7days	316
CDRMaximumDiskSpaceExceeded	316
CiscoDirSyncProcessFailToStart	317
CodeRedEntry	318
CodeYellowEntry	318
CoreDumpFileFound	319
DChannelOOS	320
DUPLEX_MISMATCH	320
ErrorChangeNotifyClientBlock	321
LogPartitionHighWaterMarkExceeded	322
MaxCallsReached	322
MGCPGatewayLostComm	323
StationTCPInitError	323
TCPSetupToIMEFailed	324
TimerThreadSlowed	325
TestAlarmCritical	325
Error-Level Alarms	326
ANNDeviceRecoveryCreateFailed	326
AwaitingResponseFromPDPTimeout	327
BadCDRFileFound	327
BDIApplicationError	328
BDIOverloaded	329
CARSchedulerJobError	329
CARSchedulerJobFailed	330
CCDIPReachableTimeOut	331
CCDPSTNFailOverDurationTimeOut	331
CDRAgentSendFileFailed	332
CDRAgentSendFileFailureContinues	333
CDRFileDeliveryFailed	334
CDRFileDeliveryFailureContinues	334
CFBDeviceRecoveryCreateFailed	335
CiscoDhcpdFailure	336
CiscoDirSyncProcessFailedRetry	336
CiscoDirSyncProcessFailedNoRetry	337
CiscoDirSyncProcessConnectionFailed	337
CiscoDirSyncDBAccessFailure	337
CiscoLicenseManagerDown	338
CiscoLicenseRequestFailed	338
CiscoLicenseDataStoreError	339
CiscoLicenseInternalError	339
CiscoLicenseFileError	339
CLM_MsgIntChkError	340
CLM_UnrecognizedHost	340
ConfigItAllBuildFilesFailed	341
ConfigItAllReadConfigurationFailed	341
ConfigThreadBuildFileFailed	342
ConfigThreadCNCMGrpBuildFileFailed	343
ConfigThreadCNGrpBuildFileFailed	343
ConfigThreadReadConfigurationFailed	344
ConfigThreadUnknownExceptionCaught	344
ConflictingDataIE	345
ConnectionFailure	345
ConnectionFailureToPDP	348
CNFFBuffWriteToFilefopenfailed	348
CNFFBuffWriteToFilefwritefailed	349
CtiProviderOpenFailure	350
DBLGetVersionInfoError	352
DeviceTypeMismatch	352
DbInfoCorrupt	356
DbInfoError	356
DbInfoTimeout	357
DeviceCloseMaxEventsExceeded	357
DeviceInitTimeout	358
DirSyncSchedulerFailedToUpdateNextExecTime	358
DirSyncScheduledTaskFailed	359
DirSyncSchedulerFailedToGetDBSchedules	359
DirSyncSchedulerInvalidEventReceived	360
DirSyncInvalidScheduleFound	360
DirSyncSchedulerFailedToRegisterDBEvents	360
DirSyncSchedulerEngineFailedToStart	361
DirSyncScheduleDeletionFailed	361
DirSyncScheduleUpdateFailed	362
DRFMasterAgentStartFailure	362
DRFLocalAgentStartFailure	363
DRFRestoreFailure	363
DRFInternalProcessFailure	364
DRFTruststoreMissing	365
DRFUnknownClient	365
DRFSecurityViolation	366
DRFBackupDeviceError	367
DRFTapeDeviceError	367
DRFRestoreInternalError	368
DRFMABackupComponentFailure	369
DRFMARestoreComponentFailure	369
DRFMABackupNodeDisconnect	370
DRFNoRegisteredComponent	371
DRFNoRegisteredFeature	371
DRFMARestoreNodeDisconnect	372
DRFSftpFailure	373
DRFRegistrationFailure	373
DRFBackupCancelInternalError	374
DRFLogDirAccessFailure	375
DRFFailure	375
DRFLocalDeviceError	376
DuplicateLearnedPattern	377
EMAppInitializationFailed	377
EMCCFailedInLocalCluster	378
EMServiceConnectionError	379
EndPointTransientConnection	379
EndPointUnregistered	385
ErrorChangeNotifyClientTimeout	391
ErrorParsingDirectiveFromPDP	391
ErrorReadingInstalledRPMS	392
FailureResponseFromPDP	392
FailedToReadConfig	393
FirewallMappingFailure	394
ICTCallThrottlingStart	394
IDSEngineCritical	395
IDSEngineFailure	396
IDSReplicationFailure	396
InsufficientFallbackIdentifiers	397
InvalidIPNetPattern	398
InvalidPortHandle	398
IPMAApplicationError	399
IPMAOverloaded	399
IPMAFilteringDown	400
IPv6InterfaceNotInstalled	400
kANNDeviceRecordNotFound	401
kCFBDeviceRecordNotFound	402
kCreateAudioSourcesFailed	402
kCreateControlFailed	403
kDbConnectionFailed	404
kIPVMSDeviceDriverNotFound	404
kIpVmsMgrNoLocalHostName	405
kIpVmsMgrNoLocalNetworkIPAddr	405
kIPVMSMgrWrongDriverVersion	406
kMOHTFTPGoRequestFailed	407
kPWavMgrThreadxFailed	407
kReadCfgUserLocaleEnterpriseSvcParm	408
kRequestedANNStreamsFailed	408
LostConnectionToSAFForwarder	409
MultipleSIPTrunksToSamePeerAndLocalPort	409
NodeNotTrusted	410
NumDevRegExceeded	410
PublishFailedOverQuota	411
ReadConfigurationUnknownException	412
ReadingFileFailure	412
RsvpNoMoreResourcesAvailable	413
RTMT_ALERT	415
RTMT-ERROR-ALERT	416
SAFForwarderError	416
SAFResponderError	419
ScheduledCollectionError	420
SerialPortGetStatusError	420
SerialPortSetStatusError	421
ServiceActivationFailed	422
ServiceDeactivationFailed	422
ServiceFailed	423
ServiceStartFailed	423
ServiceStopFailed	423
ServiceExceededMaxRestarts	424
SIPNormalizationResourceWarning	424
SIPNormalizationScriptError	426
SIPTrunkOOS	427
SparePartitionLowWaterMarkExceeded	430
SystemResourceError	430
TestAlarmError	431
ThreadPoolProxyUnknownException	431
UnableToRegisterwithCallManagerService	432
UserLoginFailed	432
WritingFileFailure	433
WDApplicationError	433
WDOverloaded	434
Warning-Level Alarms	434
AnnunciatorNoMoreResourcesAvailable	434
ApplicationConnectionDropped	435
ApplicationConnectionError	436
authAdminLock	437
AuthenticationFailed	437

Match case Limit results 1 per page

3-38

Cisco Unified Communications Manager Managed Services Guide

OL-22523-01

Chapter 3

Managing and Monitoring the Health of Cisco Unified Communications Manager Systems

General Health and Troubleshooting Tips

•

Cisco RIS Data Collector PerfMonLog is disabled by default in Cisco Unified CM Release 5.x and

enabled by default in Cisco Unified CM Release 6.0.

Cisco CallManager Service Stops Responding

When the Cisco CallManager service stops responding, the following message displays in the System

Event log:

The Cisco CallManager service terminated unexpectedly.

It has done this 1 time. The following corrective action

will be taken in 60000 ms. Restart the service.

Other messages you may see in this situation:

Timeout 3000 milliseconds waiting for

Cisco CallManager service to connect.

The Cisco Communications Manager failed to start due to the following error:

The service did not respond to the start or control request in a timely fashion.

At this time when devices such as the Cisco Unified IP Phones and gateways, unregister from the Cisco

Unified Communications Manager, users receive delayed dial tone, and/or the Cisco Unified

Communications Manager server freezes due to high CPU usage. For event log messages that are not

included here, view the Cisco Unified Communications Manager Event Logs.

Possible Cause

The Cisco CallManager service can stop responding because the service does not have enough resources

such as CPU or memory to function. Generally, the CPU utilization in the server is 100 percent at that

time.

Recommended Action

Depending on what type of interruption you experience, you will need to gather different data that will

help determine the root cause of the interruption.

Use the following procedure if a lack of resources interruption occurs.

Procedure

Step 1

Collect Cisco CallManager traces 15 minutes before and after the interruption.

Step 2

Collect SDL traces 15 minutes before and after the interruption.

Step 3

Collect perfmon traces if available.

Step 4

If the traces are not available, start collecting the perfmon traces and track memory and CPU usage for

each process that is running on the server. These will help in the event of another lack of resources

interruption.