익스트림 스위치 CPU 점유문제.

익스트림 스위치 CPU 점유문제를 확인하고 해결한 방법.

모델: Extreme X430-48t
이상 증상: CPU 0% IDLE. 응답시간이 길어지는 현상 발생

top 명령으로 리소스 사용량을 확인해보니, 아래처럼, 0.0% idle, bcmRX, tbcm_msm_txX 프로세스의 CPU 점유율이 높다.

Switch.1 # top
Mem: 193584K used, 61432K free, 0K shrd, 0K buff, 78476K cached
CPU:  0.0% usr 20.0% sys  0.0% nic  0.0% idle  0.0% io  0.0% irq 80.0% sirq
Load average: 7.42 7.58 7.38 4/152 1532
  PID  PPID USER     STAT   RSS %MEM CPU %CPU COMMAND
 1110     2 root     RW       0  0.0   0 42.1 [bcmRX]
 1107     2 root     DW<      0  0.0   0 42.1 [tbcm_msm_tx1]
 1532  1531 root     R      648  0.2   0 10.5 top -d 3 
 1105     2 root     DW<      0  0.0   0  5.2 [tbcm_msm_tx0]
 1262     1 root     S    12224  4.7   0  0.0 ./cliMaster 
 1240     1 root     S <  10440  4.0   0  0.0 ./hal 
 1333     1 root     S     5760  2.2   0  0.0 ./netTools 
 1342     1 root     S     5296  2.0   0  0.0 ./etmon 
 1269     1 root     S     4816  1.8   0  0.0 ./snmpMaster 
 1358     1 root     S     4688  1.8   0  0.0 ./xmld 
 1271     1 root     S     4100  1.6   0  0.0 ./snmpSubagent 
 1371     1 root     S     4064  1.5   0  0.0 ./idMgr 
 1282     1 root     S     3964  1.5   0  0.0 ./aaa -t random 
 1320     1 root     S     3800  1.4   0  0.0 ./rtmgr update 
 1236     1 root     S     3416  1.3   0  0.0 ./emsServer 
 1326     1 root     S     3376  1.3   0  0.0 ./mcmgr 
 1286     1 root     S     3332  1.3   0  0.0 ./vlan 
 1288     1 root     S     3060  1.2   0  0.0 ./fdb 
 1330     1 root     S     2964  1.1   0  0.0 ./acl 
 1344     1 root     S     2720  1.0   0  0.0 ./thttpd 
 1264     1 root     S     2560  1.0   0  0.0 ./cfgmgr 
 1310     1 root     S     2532  0.9   0  0.0 ./stp 
 1376     1 root     S     2528  0.9   0  0.0 ./erps 
 1366     1 root     S     2496  0.9   0  0.0 ./dot1ag 
 1383     1 root     S     2420  0.9   0  0.0 ./exsshd 
 1303     1 root     S     2400  0.9   0  0.0 ./eaps 
  947     1 root     S     2384  0.9   0  0.0 /exos/bin/epm -t 40 -f /exos/config/epmrc.L2 -d /exos/config/epmdprc 
 1335     1 root     S     2368  0.9   0  0.0 ./netLogin 
 1360     1 root     S     2316  0.9   0  0.0 ./xmlc 
 1305     1 root     S     2284  0.8   0  0.0 ./esrp 
 1339     1 root     S     2244  0.8   0  0.0 ./telnetd -e 
 1378     1 root     S <   2240  0.8   0  0.0 ./mrp 
 1299     1 root     S     2216  0.8   0  0.0 ./lldp 
 1362     1 root     S     2212  0.8   0  0.0 ./ipSecurity 
 1337     1 root     S     2200  0.8   0  0.0 ./techSupport 
 1364     1 root     S     2168  0.8   0  0.0 ./upm 
 1352     1 root     S     2132  0.8   0  0.0 ./poe 
 1297     1 root     S     2124  0.8   0  0.0 ./edp 
 1290     1 root     S     2116  0.8   0  0.0 ./elrp 
 1301     1 root     S     2112  0.8   0  0.0 ./lacp 
 1368     1 root     S     2056  0.8   0  0.0 ./hclag 
 1292     1 root     R     2000  0.7   0  0.0 ./elsm 
 1238     1 root     S     1948  0.7   0  0.0 ./devmgr 

bcmRX, tbcm_msm_tx1의 높은 CPU 점유율에 관한 아래 문서를 확인했으나, 이 경우와는 관련 없는 것으로 판다.
https://gtacknowledge.extremenetworks.com/articles/Solution/High-CPU-utilization-for-the-tbcm-msm-tx1-and-bcmRX-coupled-with-CPU-congestion-on-BD8800-IO-modules

아래 문서에서 0.0% CPU idle의 경우 loop 발생가능성이 있다는 언급을 보고, 아래처럼 포트 사용율을 확인했다. 일단, 아래처럼 27, 30번 포트가 의심스러움.

* Switch.2 # sh ports util band
Port     Link    Link   Rx             Peak Rx       Tx            Peak Tx
         State   Speed  % bandwidth    % bandwidth   % bandwidth   % bandwidth
================================================================================
1         R       0         0.00          0.00          0.00            0.00
2         A       100       0.00          0.18          0.00            3.33
3         A       100       0.00          0.49          0.00            2.27
4         A       100       0.00          0.12          0.00            2.33
5         R       0         0.00          0.00          0.00            0.00
6         A       100       0.01          0.03          0.02            0.95
7         R       0         0.00          0.00          0.00            0.00
8         A       100       0.03          0.04          1.28           57.47
9         A       100       0.00          0.15          0.00            1.42
10        A       100       0.00          0.00          0.00            0.57
11        R       0         0.00          0.00          0.00            0.00
12        A       100       0.00          0.00          0.00            0.57
13        R       0         0.00          0.00          0.00            0.00
14        A       100       0.03          0.04          2.35            3.63
15        A       100       0.00          0.02          0.00            0.57
16        A       100       0.02          0.14          0.33            8.06
17        A       100       0.00          0.02          0.00            1.74
18        A       100       0.00          0.03          0.00            0.59
19        R       0         0.00          0.00          0.00            0.00
20        A       100       0.00          0.02          0.00            0.57
21        A       100       0.00          0.00          0.00            0.57
22        R       0         0.00          0.00          0.00            0.00
23        R       0         0.00          0.00          0.00            0.00
24        R       0         0.00          0.00          0.00            0.00
25        A       1000      0.00          0.00          0.00            0.06
26        A       100       0.00          0.13          0.00            4.78
27        A       100       0.59         21.69          0.08           98.81
28        A       100       0.00          0.06          0.00            1.07
29        R       0         0.00          0.00          0.00            0.00
30        R       0         0.00        100.00          0.00            2.84
31        A       100       0.00          0.00          0.00            0.57
32        R       0         0.00          0.00          0.00            0.00
33        A       100       0.00          0.00          0.00            0.57
34        R       0         0.00          0.00          0.00            0.00
35        A       100       0.00          0.03          0.00            1.55
36        A       100       0.00          0.01          0.00            0.57
37        A       100       0.00          0.04          0.01            3.67
38        R       0         0.00          0.00          0.00            0.00
39        A       100       0.00          0.00          0.00            1.49
40        A       100       0.00          0.01          0.00            2.49
41        R       0         0.00          0.00          0.00            0.00
42        R       0         0.00          0.00          0.00            0.00
43        R       0         0.00          0.00          0.00            0.00
44        R       0         0.00          0.00          0.00            0.00
45        R       0         0.00          0.00          0.00            0.00
46        R       0         0.00          0.00          0.00            0.00
47        R       0         0.00          0.00          0.00            0.00
48        R       0         0.00          0.00          0.00            0.00
49        R       0         0.00          0.00          0.00            0.00
50        R       0         0.00          0.00          0.00            0.00
51        R       0         0.00          0.00          0.00            0.00
52        A       1000      0.40          4.22          0.07            7.44
================================================================================
          > indicates Port Display Name truncated past 8 characters
          Link State: A-Active, R-Ready, NP-Port Not Present, L-Loopback

27번 포트를 먼저 disable 해 보았으나, 증상은 개선되지 않아서, 30번 포트를 disable로 변경.

30번 포트 disable 이후, 아래처럼 CPU 81.5% idle로 이상 증상 사라졌으며, 점유율 높았던 두 프로세스도 점유율이 낮아져서 보이지 않게 되었다.

Mem: 200696K used, 54320K free, 0K shrd, 0K buff, 79864K cached
CPU:  2.9% usr 12.1% sys  0.0% nic 81.5% idle  0.0% io  2.6% irq  0.6% sirq
Load average: 4.46 5.15 6.49 4/152 1562
  PID  PPID USER     STAT   RSS %MEM CPU %CPU COMMAND
 1072     2 root     SW<      0  0.0   0  3.9 [bcmLINK.1]
 1070     2 root     SW<      0  0.0   0  3.6 [bcmLINK.0]
 1288     1 root     S     3128  1.2   0  1.3 ./fdb 
 1282     1 root     S     3992  1.5   0  0.9 ./aaa -t random 
 1290     1 root     S     2120  0.8   0  0.9 ./elrp 
 1562  1561 root     R      800  0.3   0  0.9 top -d 3 
 1526     2 root     SW<      0  0.0   0  0.9 [bcmCNTR.1]
 1525     2 root     SW<      0  0.0   0  0.9 [bcmCNTR.0]
 1240     1 root     S <  10492  4.1   0  0.6 ./hal 
 1330     1 root     S     3036  1.1   0  0.3 ./acl 
 1360     1 root     R     2356  0.9   0  0.3 ./xmlc 
  946   945 root     S     1768  0.6   0  0.3 /exos/bin/exsh 
 1242     1 root     S     1620  0.6   0  0.3 ./nodemgr 
 1262     1 root     S    17848  6.9   0  0.0 ./cliMaster 
 1333     1 root     S     5880  2.3   0  0.0 ./netTools 
 1342     1 root     S     5324  2.0   0  0.0 ./etmon 
 1269     1 root     S     4872  1.9   0  0.0 ./snmpMaster 
 1358     1 root     S     4688  1.8   0  0.0 ./xmld 
 1371     1 root     S     4112  1.6   0  0.0 ./idMgr 
 1271     1 root     S     4100  1.6   0  0.0 ./snmpSubagent 
 1320     1 root     S     3856  1.5   0  0.0 ./rtmgr update 
 1286     1 root     S     3604  1.4   0  0.0 ./vlan 
 1236     1 root     S     3420  1.3   0  0.0 ./emsServer 
 1326     1 root     S     3404  1.3   0  0.0 ./mcmgr 
 1264     1 root     S     2852  1.1   0  0.0 ./cfgmgr 

스위치 포트의 카운터를 초기화한 후, 포트 사용율 점검.

* Switch.4 # clear counters ports 
* Switch.5 # sh ports utilization bandwidth 
Port     Link    Link   Rx             Peak Rx       Tx            Peak Tx
         State   Speed  % bandwidth    % bandwidth   % bandwidth   % bandwidth
================================================================================
1         R       0         0.00          0.00          0.00            0.00
2         A       100       0.00          0.00          0.00            0.00
3         A       100       0.01          0.01          0.01            0.01
4         A       100       0.00          0.00          0.00            0.00
5         R       0         0.00          0.00          0.00            0.00
6         A       100       0.00          0.00          0.00            0.00
7         R       0         0.00          0.00          0.00            0.00
8         A       100       0.02          0.04          1.89            2.30
9         A       100       0.03          0.04          0.09            0.13
10        A       100       0.00          0.00          0.00            0.00
11        R       0         0.00          0.00          0.00            0.00
12        A       100       0.00          0.00          0.00            0.00
13        R       0         0.00          0.00          0.00            0.00
14        A       100       0.06          0.11          6.24            7.35
15        A       100       0.00          0.00          0.00            0.00
16        A       100       0.00          0.00          0.01            0.01
17        A       100       0.00          0.00          0.00            0.00
18        A       100       0.00          0.00          0.00            0.00
19        R       0         0.00          0.00          0.00            0.00
20        A       100       0.14          0.14          5.23            5.23
21        A       100       0.00          0.00          0.00            0.00
22        R       0         0.00          0.00          0.00            0.00
23        R       0         0.00          0.00          0.00            0.00
24        R       0         0.00          0.00          0.00            0.00
25        A       1000      0.00          0.00          0.00            0.00
26        A       100       0.00          0.00          0.00            0.00
27        A       100       0.04          0.14          3.78            8.48
28        A       100       0.00          0.00          0.00            0.00
29        R       0         0.00          0.00          0.00            0.00
30        R       0         0.00          0.00          0.00            0.00
31        A       100       0.00          0.00          0.00            0.00
32        R       0         0.00          0.00          0.00            0.00
33        A       100       0.00          0.00          0.00            0.00
34        R       0         0.00          0.00          0.00            0.00
35        A       100       0.00          0.00          0.00            0.00
36        A       100       0.00          0.00          0.00            0.00
37        A       100       0.33          0.47          4.81           27.71
38        R       0         0.00          0.00          0.00            0.00
39        A       100       0.00          0.00          0.00            0.00
40        A       100       0.00          0.00          0.00            0.01
41        R       0         0.00          0.00          0.00            0.00
42        R       0         0.00          0.00          0.00            0.00
43        R       0         0.00          0.00          0.00            0.00
44        R       0         0.00          0.00          0.00            0.00
45        R       0         0.00          0.00          0.00            0.00
46        R       0         0.00          0.00          0.00            0.00
47        R       0         0.00          0.00          0.00            0.00
48        R       0         0.00          0.00          0.00            0.00
49        R       0         0.00          0.00          0.00            0.00
50        R       0         0.00          0.00          0.00            0.00
51        R       0         0.00          0.00          0.00            0.00
52        A       1000      1.55          4.63          0.05            0.08
================================================================================
          > indicates Port Display Name truncated past 8 characters
          Link State: A-Active, R-Ready, NP-Port Not Present, L-Loopback
* Switch.9 # 

답글 남기기

Your email address will not be published.

%d bloggers like this: