Cisco Nexus 7000 수퍼바이저 모듈 powered down.
Active supervisor 상태가 powered-dn 으로 바뀌고 standby 모듈이 active 상태가 되었으며, 경고등이 점등된 상태.
vdc-2 에서는 아래와 같은 로그를 확인 할 수 있었다.
2023 Aug 21 13:30:24 Backbone_2-Main %PLATFORM-2-MOD_PWRDN: Module 5 powered down (Serial number ) 2023 Aug 21 13:30:24 Backbone_2-Main %PLATFORM-5-MOD_STATUS: Module 5 current-status is MOD_STATUS_CONFIGPOWERED_DOWN 2023 Aug 21 13:30:24 Backbone_2-Main %PLATFORM-5-MOD_STATUS: Module 5 current-status is MOD_STATUS_POWERED_DOWN 2023 Aug 21 13:30:25 Backbone_2-Main %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number )
admin vdc에서 아래와 같은 작업을 진행 하였다.
1. 모듈 상태 확인.
show module 명령어로 모듈상태를 보면, 5번 모듈이 powered down 상태인 것을 볼 수 있다. Backbone_2# show module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 3 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 5 0 Supervisor Module-2 N7K-SUP2E powered-dn 6 0 Supervisor Module-2 N7K-SUP2E active * 8 12 10/40 Gbps Ethernet Module N7K-F312FQ-25 ok 10 48 1/10 Gbps BASE-T Ethernet Module N7K-F248XT-25E ok Mod Power-Status Reason --- ------------ --------------------------- 5 powered-dn Policy trigger initiated reset: Stdby has lower mem than act ive Mod Sw Hw --- -------------- ------ 1 6.2(20) 1.1 3 6.2(20) 1.1 6 6.2(20) 6.1 8 6.2(20) 1.1 10 6.2(20) 1.3 Mod MAC-Address(es) Serial-Num --- -------------------------------------- ---------- 1 50-87-89-48-68-74 to 50-87-89-48-68-a7 JAF9999ATSM 3 50-87-89-48-26-08 to 50-87-89-48-26-3b JAF9999AMDN 5 00-00-00-00-00-00 to 00-00-00-00-00-00 NA 6 50-87-89-46-09-49 to 50-87-89-46-09-5b JAF9998ANRA 8 fc-5b-39-f9-19-10 to fc-5b-39-f9-19-4b JAE999702RG 10 50-87-89-44-91-bc to 50-87-89-44-91-ef JAF1831AGHL Mod Online Diag Status --- ------------------ 1 Pass 3 Pass 6 Pass 8 Pass 10 Pass Xbar Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 0 Fabric Module 2 N7K-C7010-FAB-2 ok 2 0 Fabric Module 2 N7K-C7010-FAB-2 ok 3 0 Fabric Module 2 N7K-C7010-FAB-2 ok 4 0 Fabric Module 2 N7K-C7010-FAB-2 ok 5 0 Fabric Module 2 N7K-C7010-FAB-2 ok Xbar Sw Hw --- -------------- ------ 1 NA 1.6 2 NA 1.6 3 NA 1.6 4 NA 1.6 5 NA 1.6 Xbar MAC-Address(es) Serial-Num --- -------------------------------------- ---------- 1 NA JAF9998AGAA 2 NA JAF9997ADNR 3 NA JAF9998AGDT 4 NA JAF9998ALQF 5 NA JAF9998AMDD * this terminal session
2. admin-vdc에서 아래처럼 관련된 로그를 확일 할 수 있다.
2023 Aug 21 13:28:24 Backbone_2 %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase). 2023 Aug 21 13:28:24 Backbone_2 %SYSMGR-2-HASWITCHOVER_START: Supervisor 6 is becoming active. 2023 Aug 21 13:28:24 Backbone_2 %SYSMGR-2-SWITCHOVER_OVER: Switchover completed. 2023 Aug 21 13:28:24 Backbone_2 %COPP-2-COPP_NO_POLICY: Control-plane is unprotected. 2023 Aug 21 13:28:24 Backbone_2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operational members) 2023 Aug 21 13:28:25 Backbone_2 %VDC_MGR-5-VDC_MODULETYPE: vdc_mgr: Module type changed to None 2023 Aug 21 13:28:25 Backbone_2 %CALLHOME-2-EVENT: SUP_FAILURE 2023 Aug 21 13:28:25 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.28875 2023 Aug 21 13:28:25 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.28883 2023 Aug 21 13:28:25 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.28847 2023 Aug 21 13:28:25 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.28861 2023 Aug 21 13:28:25 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.28877 2023 Aug 21 13:28:58 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.29213 2023 Aug 21 13:29:02 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.29222 2023 Aug 21 13:29:04 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.29228 2023 Aug 21 13:29:25 Backbone_2 %PLATFORM-4-PFM_FAN_FLTR_STATUS: Fan Filter status: Present 2023 Aug 21 13:30:24 Backbone_2 %SYSMGR-2-SUP_POWERDOWN: Supervisor in slot 5 is running with less memory than active supervisor in slot 6 2023 Aug 21 13:30:24 Backbone_2 %SYSMGR-2-CONVERT_STARTUP_ABORTED: Conversion of startup-config failed. 2023 Aug 21 13:30:24 Backbone_2 %PLATFORM-2-MOD_PWRDN: Module 5 powered down (Serial number ) 2023 Aug 21 13:30:24 Backbone_2 %PLATFORM-5-MOD_STATUS: Module 5 current-statusis MOD_STATUS_CONFIGPOWERED_DOWN 2023 Aug 21 13:30:24 Backbone_2 %PLATFORM-5-MOD_STATUS: Module 5 current-statusis MOD_STATUS_POWERED_DOWN 2023 Aug 21 13:30:25 Backbone_2 %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number ) 2023 Aug 21 13:33:24 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.29663
3. 5번 모듈을 물리적으로 제거했다가 잠시후 다시 장착하였다.
이때, 콘솔에서는 아래처럼 모듈 제거와 재 장착에 대한 메시지를 확인 할 수 있다.
Backbone_2# 2023 Sep 15 13:50:17 Backbone_2 %$ VDC-1 %$ %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number ) 2023 Sep 15 13:50:17 Backbone_2-Main %$ VDC-2 %$ %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number ) 2023 Sep 15 13:50:43 Backbone_2 %$ VDC-1 %$ %PLATFORM-2-MODULE_EJECTOR_POLICY_ENABLED: All Ejectors closed for module 5. Ejector based shutdown enabled 2023 Sep 15 13:50:43 Backbone_2 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 5 detected (Serial number :unavailable) Module-Type Supervisor Module-2 Model :unavailable 2023 Sep 15 13:50:43 Backbone_2-Main %$ VDC-2 %$ %PLATFORM-2-MOD_DETECT: Module 5 detected (Serial number :unavailable) Module-Type Supervisor Module-2 Model :unavailable
4. 재부팅 후 모듈 확인하면,
모듈 5의 상태가 powered-up 상태로 보인다.
Backbone_2# show module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 3 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 5 0 Supervisor Module-2 powered-up 6 0 Supervisor Module-2 N7K-SUP2E active * 8 12 10/40 Gbps Ethernet Module N7K-F312FQ-25 ok 10 48 1/10 Gbps BASE-T Ethernet Module N7K-F248XT-25E ok
시간이 조금 지난후 다시 확인해 보면, 모듈 5의 상태가 ha-standby로 바뀐 것이 보인다.
Backbone_2# show module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 3 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 5 0 Supervisor Module-2 N7K-SUP2E ha-standby 6 0 Supervisor Module-2 N7K-SUP2E active * 8 12 10/40 Gbps Ethernet Module N7K-F312FQ-25 ok 10 48 1/10 Gbps BASE-T Ethernet Module N7K-F248XT-25E ok
5. 그냥 사용해도 되겠지만, 원래 모듈 5가 active 수퍼바이저였으므로, 5번 모듈을 active 수퍼바이저로 변경한다.
switchover명령을 내리면 active 수퍼바이저가 변경되므로, 콘솔 연결이 끊어진다.
Backbone_2# system switchover Backbone_2# User Access Verification Backbone_2 login: >>> >>>
콘솔 케이블을 5번 모듈에 다시 연결해야 한다.
User Access Verification Backbone_2 login: admin Password:
모듈 상태를 확인해 보면 5번 모듈이 active 상태로 바뀌었으며, 6번 모듈은 powered-up 상태이다.
Backbone_2# show module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 3 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 5 0 Supervisor Module-2 N7K-SUP2E active * 6 0 Supervisor Module-2 powered-up 8 12 10/40 Gbps Ethernet Module N7K-F312FQ-25 ok 10 48 1/10 Gbps BASE-T Ethernet Module N7K-F248XT-25E ok
시간이 조금 지나면, 아래와 같이 powered-up 상태가 ha-standy로 바뀐다.
Backbone_2# sho module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 3 48 1/10 Gbps Ethernet Module N7K-F248XP-25E ok 5 0 Supervisor Module-2 N7K-SUP2E active * 6 0 Supervisor Module-2 N7K-SUP2E ha-standby 8 12 10/40 Gbps Ethernet Module N7K-F312FQ-25 ok 10 48 1/10 Gbps BASE-T Ethernet Module N7K-F248XT-25E ok
* 아래는 관련된 로그이다.
vdc-2의 로그
2023 Sep 15 11:18:52 Backbone_2-Main %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on 172.30.192.121@pts/0 2023 Sep 15 13:50:15 Backbone_2-Main %PLATFORM-3-EJECTOR_STAT_CHANGED: Ejectors' status in slot 5 has changed, Top Ejector is OPEN, Bottom Ejector is CLOSE 2023 Sep 15 13:50:17 Backbone_2-Main %PLATFORM-3-EJECTOR_STAT_CHANGED: Ejectors' status in slot 5 has changed, Top Ejector is OPEN, Bottom Ejector is OPEN 2023 Sep 15 13:50:17 Backbone_2-Main %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number ) 2023 Sep 15 13:50:43 Backbone_2-Main %PLATFORM-2-MOD_DETECT: Module 5 detected (Serial number :unavailable) Module-Type Supervisor Module-2 Model :unavailable
admin vdc 로그.
2023 Sep 15 13:50:15 Backbone_2 %PLATFORM-3-EJECTOR_STAT_CHANGED: Ejectors' status in slot 5 has changed, Top Ejector is OPEN, Bottom Ejector is CLOSE 2023 Sep 15 13:50:17 Backbone_2 %PLATFORM-3-EJECTOR_STAT_CHANGED: Ejectors' status in slot 5 has changed, Top Ejector is OPEN, Bottom Ejector is OPEN 2023 Sep 15 13:50:17 Backbone_2 %PLATFORM-2-MOD_REMOVE: Module 5 removed (Serial number ) 2023 Sep 15 13:50:43 Backbone_2 %PLATFORM-2-MODULE_EJECTOR_POLICY_ENABLED: All Ejectors closed for module 5. Ejector based shutdown enabled 2023 Sep 15 13:50:43 Backbone_2 %PLATFORM-2-MOD_DETECT: Module 5 detected (Serial number :unavailable) Module-Type Supervisor Module-2 Model :unavailable 2023 Sep 15 13:52:18 Backbone_2 %USBHSD-STANDBY-2-MOUNT: logflash: online 2023 Sep 15 13:53:08 Backbone_2 %BOOTVAR-5-NEIGHBOR_UPDATE_AUTOCOPY: auto-copy supported by neighbor supervisor, starting... 2023 Sep 15 13:54:43 Backbone_2 %MODULE-5-STANDBY_SUP_OK: Supervisor 5 is standby 2023 Sep 15 13:57:51 Backbone_2 Sep 15 13:57:51 %KERN-2-SYSTEM_MSG: [ 386.483815] Switchover started by redundancy driver - kernel 2023 Sep 15 13:57:51 Backbone_2 %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase). 2023 Sep 15 13:57:51 Backbone_2 %SYSMGR-2-HASWITCHOVER_START: Supervisor 5 is becoming active. 2023 Sep 15 13:57:51 Backbone_2 %SYSMGR-2-SWITCHOVER_OVER: Switchover completed. 2023 Sep 15 13:57:52 Backbone_2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operational members) 2023 Sep 15 13:57:52 Backbone_2 %COPP-2-COPP_NO_POLICY: Control-plane is unprotected. 2023 Sep 15 13:57:52 Backbone_2 %VDC_MGR-5-VDC_MODULETYPE: vdc_mgr: Module type changed to None 2023 Sep 15 13:57:52 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.8974 2023 Sep 15 13:57:56 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.9079 2023 Sep 15 13:57:56 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.9078 2023 Sep 15 13:57:56 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.9085 2023 Sep 15 13:57:56 Backbone_2 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.9090 2023 Sep 15 13:58:52 Backbone_2 %PLATFORM-4-PFM_FAN_FLTR_STATUS: Fan Filter status: Present 2023 Sep 15 14:00:21 Backbone_2 %USBHSD-STANDBY-2-MOUNT: logflash: online 2023 Sep 15 14:00:26 Backbone_2 %BOOTVAR-5-NEIGHBOR_UPDATE_AUTOCOPY: auto-copy supported by neighbor supervisor, starting... 2023 Sep 15 14:02:01 Backbone_2 %MODULE-5-STANDBY_SUP_OK: Supervisor 6 is standby