Collection of NotesCiscoN5K with N2K FEX vPC
Collection of NotesCiscoN5K with N2K FEX vPC
Cisco

N5K with N2K FEX vPC

1. Attach FEX

1.1 Configure FEX on both N5Ks

Since vPC is not enabled yet, N5K-1 configured with FEX will attach the FEX and N5K-2 won’t.

feature fex interface Ethernet1/43 no shut switchport mode fex-fabric fex associate 100 interface Ethernet1/44 no shut switchport mode fex-fabric fex associate 101
feature fex interface Ethernet1/43 no shut switchport mode fex-fabric fex associate 100 interface Ethernet1/44 no shut switchport mode fex-fabric fex associate 101

On N5K-2, FEX state is connected. But you can’t check its details which means N5K-1 manages the attached FEX control information. If you check the FEX uplink interface SFP as a test, you can’t get the information.

N5K-2# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— — ——– Offline N2K-C2232PP-10GE SSI17230KCP — ——– Offline N2K-C2232PP-10GE SSI170505TQ N5K-2# show int fex Fabric Fabric Fex FEX Fex Fex Port Port State Uplink Model Serial ————————————————————— 100 Eth1/43 Active 6 N2K-C2232PP-10GE SSI170505TQ 101 Eth1/44 Active 6 N2K-C2232PP-10GE SSI17230KCP N5K-2# show int eth1/43 transceiver fex-fabric Ethernet1/43 : sfp information is not available

1.2 FEX Attachment State Change Process

Logs on N5K-1. N5K-2 goes through the same process but doesn’t mange the FEX.

2025 Mar 18 10:55:18 N5K-1 %FEX-5-FEX_ENABLED: FEX Enabled 2025 Mar 18 10:55:21 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 0 that is connected with Ethernet1/43 changed its status from Created to Discovered 2025 Mar 18 10:56:04 N5K-1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/43 is down(Config change) 2025 Mar 18 10:56:04 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 0 that is connected with Ethernet1/43 changed its status from Discovered to Created 2025 Mar 18 10:56:05 N5K-1 %ETHPORT-5-SPEED: Interface Ethernet1/43, operational speed changed to 10 Gbps 2025 Mar 18 10:56:05 N5K-1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/43, operational duplex mode changed to Full 2025 Mar 18 10:56:05 N5K-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/43, operational Receive Flow Control state changed to off 2025 Mar 18 10:56:05 N5K-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/43, operational Transmit Flow Control state changed to off 2025 Mar 18 10:56:08 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 0 that is connected with Ethernet1/43 changed its status from Created to Discovered 2025 Mar 18 10:56:11 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Discovered to Configured 2025 Mar 18 10:56:11 N5K-1 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/43 is down (Initializing) 2025 Mar 18 10:56:14 N5K-1 %ETHPORT-5-SPEED: Interface Ethernet1/43, operational speed changed to 10 Gbps 2025 Mar 18 10:56:14 N5K-1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/43, operational duplex mode changed to Full 2025 Mar 18 10:56:14 N5K-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/43, operational Receive Flow Control state changed to off 2025 Mar 18 10:56:14 N5K-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/43, operational Transmit Flow Control state changed to off 2025 Mar 18 10:56:14 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 0 of Fex 100 that is connected with Ethernet1/43 changed its status from Configured to Fabric Up 2025 Mar 18 10:56:15 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Fabric Up to Connecting 2025 Mar 18 10:56:15 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Connecting to Active 2025 Mar 18 10:56:15 N5K-1 %ETHPORT-5-IF_UP: Interface Ethernet1/43 is up in mode Fex Fabric

When I shut the port towards FEX, I got the following logs.

2025 Mar 19 01:35:15 N5K-1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/43 is down(Config change) 2025 Mar 19 01:35:15 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Active to Disconnected 2025 Mar 19 01:35:15 N5K-1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-100 Off-line (Serial Number SSI170505TQ) 2025 Mar 19 01:35:15 N5K-1 %PFMA-2-FEX_STATUS: Fex 100 is offline 2025 Mar 19 01:35:15 N5K-1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet100/1/1 is down (Interface removed) 2025 Mar 19 01:35:15 N5K-1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet100/1/2 is down (Interface removed) …… 2025 Mar 19 01:35:16 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Disconnected to Configured 2025 Mar 19 01:35:16 N5K-1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/43 is down (Administratively down)

The logs on the N5K-2 is as follows. It takes 32 seconds for N2K to be back online on the other N5K.

2025 Mar 19 01:27:33 N5K-2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 6 of Fex 100 that is connected with Ethernet1/43 changed its status from Active to Disconnected 2025 Mar 19 01:27:34 N5K-2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 6 of Fex 100 that is connected with Ethernet1/43 changed its status from Disconnected to Fabric Up 2025 Mar 19 01:27:35 N5K-2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 6 of Fex 100 that is connected with Ethernet1/43 changed its status from Fabric Up to Connecting 2025 Mar 19 01:27:35 N5K-2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 6 of Fex 100 that is connected with Ethernet1/43 changed its status from Connecting to Active 2025 Mar 19 01:27:40 N5K-2 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet100/1/1 is down (Transceiver Absent) 2025 Mar 19 01:27:40 N5K-2 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet100/1/2 is down (Transceiver Absent) ……

The FEX 100 is hidden from N5K-1 and shown on N5K-2.

N5K-1# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— 101 FEX0101 Online N2K-C2232PP-10GE SSI17230KCP N5K-2# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— 100 FEX0100 Online N2K-C2232PP-10GE SSI170505TQ — ——– Offline N2K-C2232PP-10GE SSI17230KCP

Now I unshut the link towards FEX on N5K-1, I got the following log.

2025 Mar 19 01:50:05 N5K-1 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/43 is admin up. 2025 Mar 19 01:50:09 N5K-1 %ETHPORT-5-SPEED: Interface Ethernet1/43, operational speed changed to 10 Gbps 2025 Mar 19 01:50:09 N5K-1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/43, operational duplex mode changed to Full 2025 Mar 19 01:50:09 N5K-1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/43, operational Receive Flow Control state changed to off 2025 Mar 19 01:50:09 N5K-1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/43, operational Transmit Flow Control state changed to off 2025 Mar 19 01:50:09 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 0 of Fex 100 that is connected with Ethernet1/43 changed its status from Configured to Fabric Up 2025 Mar 19 01:50:09 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Fabric Up to Connecting 2025 Mar 19 01:50:09 N5K-1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 5 of Fex 100 that is connected with Ethernet1/43 changed its status from Connecting to Active 2025 Mar 19 01:50:09 N5K-1 %ETHPORT-5-IF_UP: Interface Ethernet1/43 is up in mode Fex Fabric

FEX is still attached to N5K-2.

N5K-1# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— 101 FEX0101 Online N2K-C2232PP-10GE SSI17230KCP — ——– Connected N2K-C2232PP-10GE SSI170505TQ

1.3 FEX Show Command Output

Show FEX interface related information.

N5K-1# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— 100 FEX0100 Online N2K-C2232PP-10GE SSI170505TQ 101 FEX0101 Online N2K-C2232PP-10GE SSI17230KCP N5K-1# show interface fex Fabric Fabric Fex FEX Fex Fex Port Port State Uplink Model Serial ————————————————————— 100 Eth1/43 Active 5 N2K-C2232PP-10GE SSI170505TQ 101 Eth1/44 Active 5 N2K-C2232PP-10GE SSI17230KCP
N5K-1# show fex 100 detail FEX: 100 Description: FEX0100 state: Online FEX version: 7.3(13)N1(1) [Switch version: 7.3(13)N1(1)] FEX Interim version: 7.3(13)N1(1) Switch Interim version: 7.3(13)N1(1) Extender Serial: SSI170505TQ Extender Model: N2K-C2232PP-10GE, Part No: 73-12533-05 Card Id: 82, Mac Addr: 70:10:5c:6f:83:82, Num Macs: 64 Module Sw Gen: 21 [Switch Sw Gen: 21] Post level: complete Pinning-mode: static Max-links: 1 Fabric port for control traffic: Eth1/43 FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Eth1/43 – Interface Up. State: Active Fex Port State Fabric Port Eth100/1/1 Down None Eth100/1/2 Down None Eth100/1/3 Down None Eth100/1/4 Down None Eth100/1/5 Down Eth1/43 Eth100/1/6 Down None Eth100/1/7 Down None Eth100/1/8 Down None Eth100/1/9 Down None Eth100/1/10 Down None Eth100/1/11 Down None Eth100/1/12 Down None Eth100/1/13 Down None Eth100/1/14 Down None Eth100/1/15 Down None Eth100/1/16 Down None Eth100/1/17 Down None Eth100/1/18 Down None Eth100/1/19 Down None Eth100/1/20 Down None Eth100/1/21 Down None Eth100/1/22 Down None Eth100/1/23 Down None Eth100/1/24 Down None Eth100/1/25 Down None Eth100/1/26 Down None Eth100/1/27 Down None Eth100/1/28 Down None Eth100/1/29 Down None Eth100/1/30 Down None Eth100/1/31 Down None Eth100/1/32 Down None Logs: 03/18/2025 10:58:10.191892: Module register received 03/18/2025 10:58:10.193770: Image Version Mismatch 03/18/2025 10:58:10.194972: Registration response sent 03/18/2025 10:58:10.195253: Requesting satellite to download image 03/18/2025 11:03:40.660036: Image preload successful. 03/18/2025 11:03:42.054991: Deleting route to FEX 03/18/2025 11:03:42.062651: Module disconnected 03/18/2025 11:03:42.064043: Module Offline 03/18/2025 11:05:11.702730: Module register received 03/18/2025 11:05:11.705782: Registration response sent 03/18/2025 11:05:11.746490: create module inserted event. 03/18/2025 11:05:11.747440: Module Online Sequence 03/18/2025 11:05:17.382921: Module Online N5K-1# show fex 101 detail …… Fabric interface state: Eth1/44 – Interface Up. State: Active Fex Port State Fabric Port Eth101/1/6 Down Eth1/44 ……

Show the transceiver info on N5K side and N2K side. They are different if using different SFPs.

——————————————————- | N5K transceiver | <---> | N2K transceiver fex-fabric| ——————————————————- N5K-1# show int eth1/43 transceiver Ethernet1/43 transceiver is present type is SFP-10G-AOC1M name is CISCO-AVAGO part number is AFBR-2CAR01Z-CS1 revision is B1 serial number is AVP2331K0VK-B nominal bitrate is 10300 MBit/sec Link length supported for AOC is 1 m cisco id is 3 cisco extended id number is 4 N5K-1# show int eth1/43 transceiver fex-fabric Ethernet1/43 transceiver is present type is SFP-10G-AOC1M name is CISCO-AVAGO part number is AFBR-2CAR01Z-CS1 revision is B1 serial number is AVP2331K0VK-A nominal bitrate is 10300 MBit/sec Link length supported for AOC is 1 m cisco id is 3 cisco extended id number is 4

Show FEX inventory and environment information.

N5K-1# show inventory fex 100 NAME: “FEX 100 CHASSIS”, DESCR: “N2K-C2232PP-10GE CHASSIS” PID: N2K-C2232PP-10GE , VID: V03 , SN: SSI170505TQ NAME: “FEX 100 Module 1”, DESCR: “Fabric Extender Module: 32x10GE, 8x10GE Superv isor” PID: N2K-C2232PP-10GE , VID: V03 , SN: FOC17132TH4 NAME: “FEX 100 Fan 1”, DESCR: “Fabric Extender Fan module” PID: N2K-C2232-FAN-B , VID: N/A , SN: N/A NAME: “FEX 100 Power Supply 1”, DESCR: “Fabric Extender AC power supply” PID: N2200-PAC-400W-B , VID: V03 , SN: LIT231533LY NAME: “FEX 100 Power Supply 2”, DESCR: “Fabric Extender AC power supply” PID: N2200-PAC-400W-B , VID: V03 , SN: LIT231533LF N5K-1# show environment fex 100 Temperature Fex 100: —————————————————————– Module Sensor MajorThresh MinorThres CurTemp Status (Celsius) (Celsius) (Celsius) —————————————————————– 1 Center 90 80 29 ok 1 Fan-side 90 80 34 ok 1 Port-side 90 80 37 ok 1 Die-1 125 100 44 ok 1 Control-1 70 56 33 ok Fan Fex: 100: —————————————————— Fan Model Hw Status —————————————————— Chassis N2K-C2232-FAN-B — ok PS-1 N2200-PAC-400W-B — ok PS-2 N2200-PAC-400W-B — no power source Power Supply Fex 100: ————————————————————————— Voltage: 12 Volts —————————————————– PS Model Power Power Status (Watts) (Amp) —————————————————– 1 N2200-PAC-400W-B 396.00 33.00 ok 2 — — — fail/shutdown Mod Model Power Power Power Power Status Requested Requested Allocated Allocated (Watts) (Amp) (Watts) (Amp) — ——————- ——- ———- ——— ———- ———- 1 N2K-C2232PP-10GE 88.80 7.40 88.80 7.40 powered-up Power Usage Summary: ——————– Power Supply redundancy mode: redundant Power Supply redundancy operational mode: Non-redundant

2. Form vPC

2.1 Configure vPC between N5Ks.

feature vpc feature lacp vpc domain 50 peer-switch role priority 15 peer-keepalive destination 192.168.1.1 source 192.168.1.2 vrf default delay restore 150 peer-gateway ip arp synchronize interface Ethernet1/45 no shut description vPC Keepalive to N5K-1 no switchport ip address 192.168.1.2/30 interface Ethernet1/46 no shut description vPC peerlink to N5K-1 switchport switchport mode trunk channel-group 12 mode active interface port-channel12 description vPC peerlink to N5K-1 switchport mode trunk spanning-tree port type network vpc peer-link

The vPC status is as follows.

N5K-1# show vpc Legend: (*) – local vPC is down, forwarding via vPC peer-link vPC domain id : 50 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 consistency status : success vPC role : primary Number of vPCs configured : 0 Peer Gateway : Enabled Peer gateway excluded VLANs : – Dual-active excluded VLANs : – Graceful Consistency Check : Enabled Operational Layer3 Peer-router : Disabled Auto-recovery status : Enabled (timeout = 240 seconds) vPC Peer-link status ——————————————————————— id Port Status Active vlans — —- —— ————————————————– 1 Po12 up 1

2.2 Configure vPC for FEX

Configure the following on both N5Ks.

interface port-channel100 switchport mode fex-fabric no lacp suspend-individual fex associate 100 vpc 100 interface port-channel101 switchport mode fex-fabric no lacp suspend-individual fex associate 101 vpc 101 interface Ethernet1/43 channel-group 100 interface Ethernet1/44 channel-group 101

Now N2Ks can be managed from either of the N5Ks.

N5K-1# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— 100 FEX0100 Online N2K-C2232PP-10GE SSI170505TQ 101 FEX0101 Online N2K-C2232PP-10GE SSI17230KCP N5K-2# show fex FEX FEX FEX FEX Fex Number Description State Model Serial ———————————————————————— 100 FEX0100 Online N2K-C2232PP-10GE SSI170505TQ 101 FEX0101 Online N2K-C2232PP-10GE SSI17230KCP

I got the following log when I make the configuration change.

2025 Mar 19 02:57:23 N5K-2 %$ VDC-1 %$ %FEX-2-FEX_PORT_STATUS_CRIT: Uplink-ID 6 of Fex 100 that is connected with Ethernet1/43 changed its status from Fabric Up to Incompatible-Topology 2025 Mar 18 02:57:25 N5K-2 %$ VDC-1 %$ %SATCTRL-FEX100-2-SATCTRL: FEX-100 Module 1: Cold boot 2025 Mar 19 02:58:16 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online 2025 Mar 19 02:58:16 N5K-2 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-100 On-line 2025 Mar 19 02:58:16 N5K-2 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 100 is online

Now the FEX interfaces are in the show vpc output.

N5K-1# show vpc Legend: (*) – local vPC is down, forwarding via vPC peer-link vPC domain id : 50 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 consistency status : success vPC role : primary Number of vPCs configured : 66 Peer Gateway : Enabled Peer gateway excluded VLANs : – Dual-active excluded VLANs : – Graceful Consistency Check : Enabled Operational Layer3 Peer-router : Disabled Auto-recovery status : Enabled (timeout = 240 seconds) vPC Peer-link status ——————————————————————— id Port Status Active vlans — —- —— ————————————————– 1 Po12 up 1 vPC status —————————————————————————- id Port Status Consistency Reason Active vlans —— ———– —— ———– ————————– ———– 100 Po100 up success success – 101 Po101 up success success – 101376 Eth100/1/1 down* Not Consistency Check Not – Applicable Performed 101377 Eth100/1/2 down* Not Consistency Check Not – Applicable Performed 101378 Eth100/1/3 down* Not Consistency Check Not – ……

2.3 Configure Host vPC

Configure VLAN, SVI interface, HSRP and port-channel towards Host.

feature interface-vlan feature hsrp vlan 120 interface Vlan120 no shutdown ip address 10.1.120.2/24 hsrp version 2 hsrp 120 preempt priority 105 ip 10.1.120.1 interface Ethernet100/1/5 no shut switchport switchport mode access switchport access vlan 120 spanning-tree port type edge channel-group 120 mode active interface port-channel120 switchport access vlan 120 spanning-tree port type edge

The status of Po120 is as follows:

N5K-1# show port-channel summary | in 120 120 Po120(SD) Eth LACP Eth100/1/5(D) N5K-1# show vpc | in 120 1 Po12 up 1,120 262263 Po120 down* Not Consistency Check Not – N5K-1# show int po120 port-channel120 is down (No operational members) vPC Status: Down, vPC number: 262263 [packets forwarded via vPC peer-link] Hardware: Port-Channel, address: 0000.0000.0000 (bia 0000.0000.0000) MTU 1500 bytes, BW 100000 Kbit, DLY 10 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, medium is broadcast Port mode is access auto-duplex, 10 Gb/s Input flow-control is off, output flow-control is on Switchport monitor is off EtherType is 0x8100 Members in this channel: Eth100/1/5 ……

If I configure vPC under Po120, I got this error.

N5K-1(config-if)# vpc 120 ERROR: Operation failed: [vPC already exists]

Leave a Reply

Your email address will not be published. Required fields are marked *