> note

A ghost OUI: why the SAN port-channel wouldn't come up between a UCS 6536 and an MDS

There are bugs you fix in five minutes and bugs that eat a whole morning because the log points you the wrong way. This is the second kind, and I'm writing it up because it is 100 % reproducible: if you ever migrate from a UCS-FI-6248 or 6300 to the 6500 series against an MDS running an old NX-OS, exactly the same thing will happen to you.

The setup#

Fabric Interconnect migration: a pair of old FIs going out, two new UCS-FI-6536 coming in, connected to the customer's MDS 9148 over an F-port-channel-trunk. Everything else about the change was clean. Hosts saw storage, compute was up, the Ethernet uplinks gave no trouble.

The only problem was the SAN, and it was an odd one. With the F-port-channel-trunk configured, the FC ports went error-disabled and the port-channel never formed. And it happened on both fabrics alike: FI-A against its MDS and FI-B against its own, same symptom. It made no difference whether I left them as port-channel members or tried a standalone port with trunking — same result, down and in error.

The log that sends you chasing ghosts#

On the FI the message was this:

%PORT-5-IF_DOWN_ERROR_DISABLED: %$VSAN ...%$ Interface fc1/30 is down (Error disabled)
pm_process_fport_vf_tagging_capability: port fc1/30 attempting vf tagging on non-cisco switch
pi_fsm_port_attr_change_init: Ifindex (fc1/30)0x101d000, Err disabled event (PI_FSM_EV_PORT_CONFIG_FAILURE)0xd7

Read it again: "attempting vf tagging on non-cisco switch". The FI was saying that the MDS, a Cisco switch through and through, was not Cisco. At that point you already know the real clue is not the obvious one.

What it was NOT (though it tempts you)#

Before getting to the bottom of it I lost a while on two of the usual suspects:

Fill-pattern. It's the classic answer for FC links that won't negotiate, especially at 8G. I reached for switchport fill-pattern IDLE out of habit. It moved nothing, because this link was negotiating at 16G and the fill-word had nothing to do with it.

Speed and cabling. Also ruled out. The optic showed healthy RX power and there were no CRC or sync errors. If it were physical layer the symptom would be different: no_light, a flapping link or a rising error count — not a dry error-disable from a configuration failure.

The detail that puts it all in order: the port had good light and still went down with a PI_FSM_EV_PORT_CONFIG_FAILURE the moment it tried to trunk. That isn't physical layer, it's protocol negotiation. And the phrase "non-cisco switch" was pointing right at it.

The real cause: an OUI the MDS didn't have in its database#

Every Cisco switch has a Switch WWN, and the first three bytes of that WWN are the OUI — the manufacturer identifier. The FI 6536 ships a new OUI, 0x70A983, released long after the NX-OS the MDS was running.

The MDS was on NX-OS 8.3(2), released in December 2018, more than four years before the first 6536 existed. Its OUI database simply did not include 0x70A983. The result: when the FI tried to form the F-port-channel-trunk, the MDS looked at the peer's OUI, didn't recognise it, and treated it as a foreign switch. Hence the "non-cisco switch" and the error-disable.

I confirmed it in 30 seconds. On the FI:

connect nxos a
show wwn switch

20:00:70:a9:83:14:55:d4
      └──┬──┘
       OUI = 0x70A983

And on the MDS, checking whether that OUI existed in its table:

show wwn oui | inc 70a983

Empty. The MDS had no idea who that manufacturer was.

The fix#

NX-OS lets you add the OUI to the MDS by hand. The decisive line is a single one, wwn oui, but I applied it together with the trunk features so as to leave the port-channel complete and force the renegotiation:

config t
wwn oui 0x70a983
!
interface port-channel 1
  shutdown
  no shutdown
end
!
copy running-config startup-config

The port-channel flap at the end is what forces a renegotiation with the OUI already loaded. The moment I ran no shutdown, the PC came up and stayed up. Case closed on fabric B.

Fabric A was identical, and here's a useful detail: both FIs in the pair came from the same batch, so the OUI was the same (0x70A983). Only the chassis-unique identifier changes (14:55:d4 vs 14:e4:44), not the OUI. Even so, it's worth checking with show wwn switch before applying, because if the batch were different the value changes and you save yourself a pointless debug.

The real fix: upgrade NX-OS#

Adding the OUI by hand works and is stable, but it's a patch. The permanent fix is upgrading the MDS to a version that already carries the OUI in its database. Cisco started updating the OUI table around 8.4(2c), and the recommended release for living alongside fifth-generation FIs is NX-OS 9.4(3b). Cisco's official compatibility matrix for FI vs MDS asks for a minimum of 9.2 for the 6536 and recommends 9.4. 8.3(2) doesn't even appear as interoperable.

If you're going to leave the manual wwn oui in place for now, document it in the MOP, because a future supervisor replacement or a reload with a clean config will wipe it out.

Why it's worth writing down#

This case normally ends in a multi-day TAC, not because it's hard to fix but because the log lies to you elegantly. The "non-cisco switch" sends half the world off to check zoning, optics or fill-word, when the problem is three bytes further up.

If it lands on you, start by looking at the new switch's OUI and check whether the far end knows it. That's the first thing, not the last.

For anyone who wants the sources, the IDs documenting this family of problems are CSCvk63036, CSCwn74098 and CSCuz11494, plus Dell KB 000291922 and Cisco TAC doc 217690.

← all notes