cpu @ 99%

[WILLIAM STEGMAN]

I have a 4510 that I recently put into production and its cpu is pegged at 99 constantly. A show proc cpu sort shows
Cat4k Mgmt LoPri to be the main culprit. A sh platform health reveals 2 processes that looked to be associated:

Code: Select all

K5L3Unicast Adj Tabl   2.00  31.49     15     11  100  500   31  33   33  3822:10
K5 L2 Hardware Addre   2.00  59.05     20     14  100  500   71  56   50  5837:19

HBG-DataCenter-4510R(config)#do sh proc cpu sorted

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  54   620006660  52990203      11700 89.28% 89.03% 88.48%   0 Cat4k Mgmt LoPri
  53    61832668 111544261        554  8.95%  8.90%  9.13%   0 Cat4k Mgmt HiPri
120     2164724   5933214        364  0.23%  0.31%  0.31%   0 Spanning Tree
  38     2478104    863775       2868  0.23%  0.25%  0.23%   0 IDB Work
114      904608   2828492        319  0.15%  0.13%  0.13%   0 IP Input
107      743744   2327469        319  0.15%  0.12%  0.08%   0 CDP Protocol
163        2880      1455       1979  0.07%  0.10%  0.41%   1 SSH Process
  14      982968   4783695        205  0.07%  0.11%  0.11%   0 ARP Input
183      107260   1481823         72  0.07%  0.01%  0.00%   0 VLAN Manager
  62      173040    282995        611  0.07%  0.02%  0.00%   0 Compute load avg
182      336500  18595895         18  0.07%  0.05%  0.07%   0 PM Callback
155      178620   5754473         31  0.07%  0.03%  0.02%   0 CEF: IPv4 proces
  13           0        10          0  0.00%  0.00%  0.00%   0 ifIndex Receive
  12           0         1          0  0.00%  0.00%  0.00%   0 Crash writer
  15           0         1          0  0.00%  0.00%  0.00%   0 CEF MIB API
  11           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  10          96         5      19200  0.00%  0.00%  0.00%   0 PrstVbl
   9           0         2          0  0.00%  0.00%  0.00%   0 Timers
   8           0       181          0  0.00%  0.00%  0.00%   0 Pool Manager
  20           4       397         10  0.00%  0.00%  0.00%   0 EEM ED Syslog
  16           4        10        400  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

sh platform cpu pack stat all

Packets Dropped In Processing by CPU event


Event             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Unknown                              0         0         0         0          0
Sa Miss                      203154944       418       482      1648        953

Any ideas why this?
thank you,
Bill

[glen_grant]

Take a look at this page and see if this helps with troubleshooting , if not a TAC case may need to be opened.

http://www.cisco.com/en/US/products/hw/ ... ef15.shtml

[WILLIAM STEGMAN]

I found an oracle listener client using the command below. It's interfaces are bonded and somehow messing up the switch's cpu.

Switch#debug platform packet all receive buffer
platform packet debugging is on
Switch#show platform cpu packet buffered
Total Received Packets Buffered: 36


thank you Glen

[Lucien Avramov]

CSCeb21247 Cat4k Sup4 high cpu problem in Cat4k Mgmt LoPri process

[WILLIAM STEGMAN]

I don't think it's a bug, I'm running Version 12.2(53)SG. Do you know what these processes might be associated with?
I am running EIGRP with a couple distribution switches as neighbors.

Code: Select all

K5L3Unicast Adj Tabl   2.00  31.49     15     11  100  500   31  33   33  3822:10
K5 L2 Hardware Addre   2.00  59.05     20     14  100  500   71  56   50  5837:19

[Lucien Avramov]

Ok now that I know your version, you are not impacted by this.

The process which has consumed the highest CPU util. was "K5 L2 Hardware Addre" and "K5L3Unicast Adj Tabl".

Process "K5 L2 Hardware Addre" is active, when new MAC addresses are learned and subsequently adjacency table has to be rewritten as well, hence "K5L3Unicast Adj Tabl" is active.

If a packet is received with an unknown source MAC address, a copy of that packet is forwarded to the CPU for MAC address learning. This is also happening, when a packet with an already known source MAC address is received on another interface. A copy is sent to the CPU and this MAC is re-learned on another interface.

In the phase of high CPU utilization, it could be observed that the traffic has been continuously received between a port channel and a gig interface for example.


I hope this clears out your questions.

[gskanth123]

Thanks all..I have not even known before

[boumendil]

Hi Lucien,

You mention that "In the phase of high CPU utilization, it could be observed that the traffic has been continuously received between a port channel and a gig interface for example.". Do you mean this is a known bug or a platform limitation.

I experience the same on multiple 4900M (either L2/L3 or pure L2 switch).


Thank you.
regards,
SB