After upgrading my NAS’s NIC to a 10Gb card, none of my zones would boot. This made sense, since they were configured to use virtual NICs that still specified the old 1Gb onboard ethernet.

So I go and use zonecfg to remove the old one and enter a new one specifying the new ixgbe0 network interface.

~# zonecfg -z myzone
zonecfg:myzone> remove net
zonecfg:myzone> add net 
zonecfg:myzone:net> set allowed-address="192.168.1.208/24"
zonecfg:myzone:net> set physical="my0"
zonecfg:myzone:net> set global-nic="ixgbe0"
zonecfg:myzone:net> set defrouter="192.168.1.1"
zonecfg:myzone:net> end
zonecfg:myzone> exit

I do this for all of my zones, and reboot to check they all turn on fine. Only 2 of my 7 zones turn on. Ruh-roh. When I try to boot one manually I at least get an error:

~# zoneadm -z myzone boot
zone 'myzone': failed to setup environment
zoneadm: zone 'myzone': call to zoneadmd failed

Sounds like ‘something’ went wrong. Pretty quickly I decide to cut my losses and just reinstall the zones that don’t work. So I uninstall one, delete its zfs filesystem, delete it’s configuration. Then I install it from fresh, all happy, until I try to boot it.

~# zoneadm -z myzone boot
zone 'myzone': failed to setup environment
zoneadm: zone 'myzone': call to zoneadmd failed

The exact same error?!? What’s going on? I completely deleted the zone!

Since all this started with swapping the NIC that that’s all that’s really changed, I figure that must be what the hold up is, maybe this new NIC can’t handle more than 2 vnics? I’ll try to create it manually to check if any errors happen.

~# dladm create-vnic -l ixgbe0 my0
dladm: vnic creation over ixgbe0 failed: object already exists

Excuse me? Already exists? I immediately list the vnics and only see 2 matching the working zones.

~# dladm show-vnic
LINK         OVER       SPEED MACADDRESS        MACADDRTYPE VID  ZONE
sab0         ixgbe0     10000 2:8:20:31:f:df    fixed       0    sabzone
rcl0         ixgbe0     10000 2:8:20:60:da:12   fixed       0    rclzone

There’s no my0 here buddy. Out of spite I try to delete it anyway:

~# dladm delete-vnic my0
~#

It does ‘something’, and now myzone boots no problem. It turns out the ghost of the previous vnic was still haunting dladm so my new ones pointing to the new ixgbe0 card couldn’t be created. The real mystery is why two of them did work no problem…

Meme of Scooby Doo being scared by zombie VNICs.