C3550, %SYS-2-MALLOCFAIL 메시지 발생.
시스코 스위치에서 대략 30초 간격으로 아래 메시지 무한 반복.
May 10 16:23:07: %SYS-2-MALLOCFAIL: Memory allocation of 1692 bytes failed from 0x153AE4, alignment 0 Pool: I/O Free: 57212 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 5 May 10 16:23:37: %SYS-2-MALLOCFAIL: Memory allocation of 1692 bytes failed from 0x153AE4, alignment 0 Pool: I/O Free: 57212 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 5 May 10 16:24:07: %SYS-2-MALLOCFAIL: Memory allocation of 1692 bytes failed from 0x153AE4, alignment 0 Pool: I/O Free: 57212 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 5
CPU 사용율도 100%에 가까움.(아래는 재부팅후라 좀 낮게 보임, 혹시 몰라 스위치 재부팅해 보았으나 동일한 메시지 발생)
Switch#sh processes cpu CPU utilization for five seconds: 98%/35%; one minute: 22%; five minutes: 47% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 3112 33006 94 0.00% 0.00% 0.00% 0 Chunk Manager 2 220 2859 76 0.00% 0.00% 0.00% 0 Load Meter 3 140 154 909 0.00% 0.00% 0.00% 0 SpanTree Helper 4 25264 2239 11283 0.00% 0.07% 0.10% 0 Check heaps 5 32580 224797 144 0.07% 0.41% 0.16% 0 Pool Manager 6 0 2 0 0.00% 0.00% 0.00% 0 Timers 7 0 1 0 0.00% 0.00% 0.00% 0 Crash writer 8 15460 42220 366 0.07% 0.21% 0.14% 0 ARP Input 9 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API 10 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT 11 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit 12 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager 13 20 5 4000 0.00% 0.00% 0.00% 0 Entity MIB API 14 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage 15 0 240 0 0.00% 0.00% 0.00% 0 IPC Dynamic Cach 16 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager 17 568 14209 39 0.00% 0.00% 0.00% 0 IPC Periodic Tim 18 0 1 0 0.00% 0.00% 0.00% 0 IPC Managed Time 19 432 14208 30 0.00% 0.00% 0.00% 0 IPC Deferred Por 20 0 2 0 0.00% 0.00% 0.00% 0 IPC Seat Manager 21 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv 22 568 14208 39 0.00% 0.00% 0.00% 0 Dynamic ARP Insp 23 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop 24 88 1519 57 0.00% 0.00% 0.00% 0 Net Input 25 460 14188 32 0.00% 0.00% 0.00% 0 GraphIt 26 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client 27 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
아래 참고 문서에 따르면, 주로 loop가 발생했을때 나오는 메시지라고 해서, 스패닝트리 확인해보니 48번 포트에서 self-looped 발생이 확인됨.
Switch#sh spanning-tree
VLAN0220
Spanning tree enabled protocol ieee
Root ID Priority 32988
Address 000b.46e2.8b00
Cost 8
Port 49 (GigabitEthernet0/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32988 (priority 32768 sys-id-ext 220)
Address 0011.20fc.b900
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 15
Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Fa0/6 Desg FWD 100 128.6 P2p
Fa0/7 Desg FWD 19 128.7 P2p
Fa0/11 Desg FWD 19 128.11 P2p
Fa0/12 Desg FWD 19 128.12 P2p
Fa0/13 Desg FWD 19 128.13 P2p
Fa0/14 Desg FWD 19 128.14 P2p
Fa0/16 Desg FWD 19 128.16 P2p
Fa0/19 Desg FWD 100 128.19 Shr
Fa0/20 Desg FWD 19 128.20 P2p
Fa0/21 Desg FWD 19 128.21 P2p
Fa0/22 Desg FWD 19 128.22 P2p
Fa0/23 Desg FWD 19 128.23 P2p
Fa0/25 Desg FWD 19 128.25 P2p
Fa0/26 Desg FWD 19 128.26 P2p
Fa0/32 Desg FWD 100 128.32 P2p
Fa0/34 Desg FWD 19 128.34 P2p
Fa0/37 Desg FWD 19 128.37 P2p
Fa0/40 Desg FWD 19 128.40 P2p
Fa0/43 Desg FWD 19 128.43 P2p
Fa0/44 Desg FWD 19 128.44 P2p
Fa0/45 Desg BLK 19 128.45 P2p self-looped
Fa0/47 Desg FWD 19 128.47 P2p
Fa0/48 Desg FWD 19 128.48 P2p
45번 포트를 shutdown 함. 이후 MALLOCFAIL 메시지 발생하지 않고, CPU 사용율도 떨어짐.
Switch# sh processes cpu CPU utilization for five seconds: 1%/0%; one minute: 22%; five minutes: 28% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 1 12432 156224 79 0.00% 0.00% 0.00% 0 Chunk Manager 2 424 5453 77 0.00% 0.00% 0.00% 0 Load Meter 3 204 192 1062 0.00% 0.00% 0.00% 0 SpanTree Helper 4 49108 4291 11444 0.00% 0.09% 0.10% 0 Check heaps 5 58248 424204 137 0.00% 0.07% 0.09% 0 Pool Manager 6 0 2 0 0.00% 0.00% 0.00% 0 Timers 7 0 1 0 0.00% 0.00% 0.00% 0 Crash writer 8 29896 80675 370 0.00% 0.04% 0.10% 0 ARP Input 9 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API 10 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT 11 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit 12 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager 13 20 5 4000 0.00% 0.00% 0.00% 0 Entity MIB API 14 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage 15 4 456 8 0.00% 0.00% 0.00% 0 IPC Dynamic Cach 16 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager 17 1024 27090 37 0.00% 0.00% 0.00% 0 IPC Periodic Tim 18 0 1 0 0.00% 0.00% 0.00% 0 IPC Managed Time 19 788 27091 29 0.00% 0.00% 0.00% 0 IPC Deferred Por 20 0 2 0 0.00% 0.00% 0.00% 0 IPC Seat Manager 21 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv 22 1192 27090 44 0.00% 0.00% 0.00% 0 Dynamic ARP Insp 23 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop 24 176 2835 62 0.00% 0.00% 0.00% 0 Net Input 25 832 27071 30 0.00% 0.00% 0.00% 0 GraphIt 26 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client 27 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd 28 9640 21134 456 0.00% 0.00% 0.00% 0 Net Background 29 0 1 0 0.00% 0.00% 0.00% 0 IDB Work 30 2300 1871 1229 0.00% 0.00% 0.00% 0 Logger 31 4364 27064 161 0.00% 0.00% 0.00% 0 TTY Background 32 5376 27079 198 0.00% 0.00% 0.00% 0 Per-Second Jobs 33 4436 463 9580 0.00% 0.00% 0.00% 0 Per-minute Jobs 34 8 7 1142 0.00% 0.00% 0.00% 0 IF-MGR control p 35 4 64 62 0.00% 0.00% 0.00% 0 IF-MGR event pro 36 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process 37 6540 5449 1200 0.00% 0.00% 0.00% 0 Compute load avg 38 1280 8165 156 0.00% 0.00% 0.00% 0 HC Counter Timer 39 480 25916 18 0.00% 0.00% 0.00% 0 Vegas Storm Cont 41 14772 461166 32 0.00% 0.00% 0.00% 0 DownWhenLooped 42 0 3 0 0.00% 0.00% 0.00% 0 Collection proce 43 2208384 601818 3669 0.23% 1.43% 1.76% 0 Vegas LED Proces 44 0 1 0 0.00% 0.00% 0.00% 0 SCQ_PROCESS 45 1828 83036 22 0.00% 0.00% 0.00% 0 RAM Access 46 23184 133687 173 0.00% 0.01% 0.00% 0 SW Frame Ager 47 96 90 1066 0.00% 0.00% 0.00% 0 VLAN Info Update 48 0 1 0 0.00% 0.00% 0.00% 0 JADE Gbit lockup 49 1716 53782 31 0.00% 0.00% 0.00% 0 L2TM Process 50 4996 53779 92 0.00% 0.00% 0.00% 0 L3TM 51 312680 77744 4021 0.39% 0.43% 0.45% 0 Vegas Statistics 52 61156 342055 178 0.00% 0.02% 0.02% 0 HMATM Learn proc 53 896 27070 33 0.00% 0.00% 0.00% 0 HMATM Age proces 54 12896 5693 2265 0.00% 0.00% 0.00% 0 VL2MM 55 0 1 0 0.00% 0.00% 0.00% 0 Vegas Bridging 56 37652 423062 88 0.00% 0.02% 0.02% 0 VegasPM 57 1392 13673 101 0.00% 0.00% 0.00% 0 VUR_MGR bg proce 58 8336 27067 307 0.00% 0.00% 0.00% 0 PI MATM Aging Pr 59 0 1 0 0.00% 0.00% 0.00% 0 Switch Backup In 60 4 455 8 0.00% 0.00% 0.00% 0 MMN bkgrd proces 61 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces 62 0 2 0 0.00% 0.00% 0.00% 0 802.1x Accountin 63 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework 64 0 25 0 0.00% 0.00% 0.00% 0 802.1x switch 65 0 1 0 0.00% 0.00% 0.00% 0 802.1x Webauth F 66 7156396 2325567 3077 0.00% 2.59% 3.01% 0 DTP Protocol 67 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework 68 20 455 43 0.00% 0.00% 0.00% 0 Port-Security 69 3696 1277 2894 0.00% 0.11% 0.18% 1 Virtual Exec 70 0 44 0 0.00% 0.00% 0.00% 0 Switch IP Host T 71 104 2727 38 0.00% 0.00% 0.00% 0 Ethchnl 72 2756 27267 101 0.00% 0.00% 0.00% 0 UDLD 73 0 1 0 0.00% 0.00% 0.00% 0 Vegas IPSG Proce 74 44 396 111 0.00% 0.00% 0.00% 0 VMATM Callback 75 52 346 150 0.00% 0.00% 0.00% 0 AAA Server 76 70196 2313137 30 0.00% 0.01% 0.00% 0 AAA ACCT Proc 77 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr 78 0 1 0 0.00% 0.00% 0.00% 0 AAA System Acct 79 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B 80 8 91 87 0.00% 0.00% 0.00% 0 IP Admin SM Proc 81 10441312 1618647 6450 0.00% 5.68% 7.89% 0 CDP Protocol 84 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R 85 4292 5439 789 0.07% 0.00% 0.00% 0 DHCP Snooping 86 0 2 0 0.00% 0.00% 0.00% 0 EAPoUDP Process 87 0 3 0 0.00% 0.00% 0.00% 0 CEF switching ba 88 45636 238616 191 0.31% 0.16% 0.17% 0 IP Input 89 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl 90 1864 52520 35 0.00% 0.00% 0.00% 0 IP ARP Track 91 0 3 0 0.00% 0.00% 0.00% 0 ADJ resolve proc 92 2904968 1823988 1592 0.00% 3.28% 4.06% 0 Spanning Tree 93 56 546 102 0.00% 0.00% 0.00% 0 CEF background p 94 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP 95 16 343 46 0.00% 0.00% 0.00% 0 Cluster L2 96 132 2726 48 0.00% 0.00% 0.00% 0 Cluster RARP 97 2503156 1513243 1654 0.00% 1.29% 1.77% 0 Cluster Base 98 224 880 254 0.00% 0.00% 0.00% 0 TCP Timer 99 104 94 1106 0.00% 0.00% 0.00% 0 TCP Protocols 100 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers 101 4 93 43 0.00% 0.00% 0.00% 0 HTTP CORE 102 0 1 0 0.00% 0.00% 0.00% 0 RARP Input 103 0 1 0 0.00% 0.00% 0.00% 0 802.1x Critical 104 380 222 1711 0.00% 0.00% 0.00% 0 L2MM 105 152 716 212 0.00% 0.00% 0.00% 0 MRD 106 7084 12499 566 0.00% 0.00% 0.00% 0 IGMPSN 107 0 1 0 0.00% 0.00% 0.00% 0 IGMPQR 108 0 2 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER 109 2216 5205 425 0.00% 0.00% 0.00% 0 DHCPD Receive 110 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers 111 0 1 0 0.00% 0.00% 0.00% 0 IP RIB Update 112 44 541 81 0.00% 0.00% 0.00% 0 Transport Port A 113 16 228 70 0.00% 0.00% 0.00% 0 DHCPD Timer 114 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA 115 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA 116 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA 118 2992 2105 1421 0.00% 0.01% 0.00% 0 TPLUS 119 848 27070 31 0.00% 0.00% 0.00% 0 Vegas EnvMon Pro 120 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF 121 0 2 0 0.00% 0.00% 0.00% 0 DHCP Security He 122 8636 106936 80 0.00% 0.00% 0.00% 0 PM Callback 123 8 35 228 0.00% 0.00% 0.00% 0 VLAN Manager 124 220 7686 28 0.00% 0.00% 0.00% 0 DHCPD Database 125 152 2182 69 0.00% 0.00% 0.00% 0 dhcp snooping sw 126 0 1 0 0.00% 0.00% 0.00% 0 VFM Queue Proces 127 20 5 4000 0.00% 0.00% 0.00% 0 VFM/VQM Merge Pr 128 4 5 800 0.00% 0.00% 0.00% 0 VFM Deferred Upd 129 12 51 235 0.00% 0.00% 0.00% 0 VTP Trap Process 131 56 144 388 0.00% 0.00% 0.00% 0 AAA SEND STOP EV 132 5808 914 6354 0.00% 0.00% 0.00% 0 Syslog Traps 133 3884 863 4500 0.00% 0.00% 0.00% 0 Syslog 134 304 5212 58 0.00% 0.00% 0.00% 0 C3550 DHCP Snoop 135 0 2 0 0.00% 0.00% 0.00% 0 STP STACK TOPOLO 136 0 2 0 0.00% 0.00% 0.00% 0 STP FAST TRANSIT 137 4 2 2000 0.00% 0.00% 0.00% 0 STP STACK TOPOLO 138 0 2 0 0.00% 0.00% 0.00% 0 CSRT RAPID TRANS 139 3220 4122 781 0.00% 0.00% 0.00% 0 IP SNMP 140 508 1430 355 0.00% 0.00% 0.00% 0 PDU DISPATCHER 141 3000 1430 2097 0.00% 0.00% 0.00% 0 SNMP ENGINE 142 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6 143 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro 144 628 160 3925 0.00% 0.00% 0.00% 0 SNMP Traps 145 1612 27415 58 0.00% 0.00% 0.00% 0 NTP