I always tell people that on the surface VMware’s products are incredibly simple and easy to use and for the most part that’s true. Anyone who’s installed an operating system can easily get a vSphere server up and running in no time at all and have a couple virtual machines up not long after. Of course with any really easy to use product the surface usability comes from an underlying system that’s incredibly complex. Those daring readers who read my last post on modifying ESXi to grant shell access to non-root users got just a taste of how complicated things can be and as you dive deeper and deeper into VMware’s world the more complicated things become.
I had a rather peculiar issue come up with one of the tools that I had developed. This tool wasn’t anything horribly complicated, all it did was change the IP address of some Windows servers and their ESXi hosts whilst switching the network over from the build VLAN to their proper production one. For the most part the tool worked as advertised and never encountered any errors, on its side at least. However people were noticing something strange about the servers that were being configured using my tool, some were coming up with a “Local Area Network 2” and “vmxnet3 Ethernet Adapter #2” as their network connection. This was strange as I wasn’t adding in any new network cards anywhere and it wasn’t happening consistently. Frustrated I dove into my code looking for answers.
After a while I figured the only place that the error could be originating from was when I was changing the server over from the build VLAN to the production one. Here’s the code, which I got from performing the same action in the VIClient proxied through Onyx, that I used to make the change:
NameValueCollection Filter = new NameValueCollection();
Filter.Add("name", "^" + ServerName);
VirtualMachine Guest = (VirtualMachine)Client.FindEntityView(typeof(VirtualMachine), null, Filter, null);
VirtualMachineConfigInfo Info = Guest.Config;
VirtualDevice NetworkCard = new VirtualDevice();
int DeviceKey = 4000;
foreach (VirtualDevice Device in Info.Hardware.Device)
{
String Identifier = Device.ToString();
if (Identifier == "VMware.Vim.VirtualVmxnet3")
{
DeviceKey = Device.Key;
NetworkCard = Device;
Console.WriteLine("INFO - Device key for network card found, ID: " + DeviceKey);
}
}
VirtualVmxnet3 Card = (VirtualVmxnet3)NetworkCard;
VirtualMachineConfigSpec Spec = new VirtualMachineConfigSpec();
Spec.DeviceChange = new VirtualDeviceConfigSpec[1];
Spec.DeviceChange[0] = new VirtualDeviceConfigSpec();
Spec.DeviceChange[0].Operation = VirtualDeviceConfigSpecOperation.edit;
Spec.DeviceChange[0].Device.Key = DeviceKey;
Spec.DeviceChange[0].Device.DeviceInfo = new VMware.Vim.Description();
Spec.DeviceChange[0].Device.DeviceInfo.Label = Card.DeviceInfo.Label;
Spec.DeviceChange[0].Device.DeviceInfo.Summary = "Build";
Spec.DeviceChange[0].Device.Backing = new VMware.Vim.VirtualEthernetCardNetworkBackingInfo();
((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).DeviceName = "Production";
((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).UseAutoDetect = false;
((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).InPassthroughMode = false;
Spec.DeviceChange[0].Device.Connectable = new VMware.Vim.VirtualDeviceConnectInfo();
Spec.DeviceChange[0].Device.Connectable.StartConnected = Card.Connectable.StartConnected;
Spec.DeviceChange[0].Device.Connectable.AllowGuestControl = Card.Connectable.AllowGuestControl;
Spec.DeviceChange[0].Device.Connectable.Connected = Card.Connectable.Connected;
Spec.DeviceChange[0].Device.Connectable.Status = Card.Connectable.Status;
Spec.DeviceChange[0].Device.ControllerKey = NetworkCard.ControllerKey;
Spec.DeviceChange[0].Device.UnitNumber = NetworkCard.UnitNumber;
((VirtualVmxnet3)Spec.DeviceChange[0].Device).AddressType = Card.AddressType;
((VirtualVmxnet3)Spec.DeviceChange[0].Device).MacAddress = Card.MacAddress;
((VirtualVmxnet3)Spec.DeviceChange[0].Device).WakeOnLanEnabled = Card.WakeOnLanEnabled;
Guest.ReconfigVM_Task(Spec);
My first inclination was that I was getting the DeviceKey wrong which is why you see me iterating through all the devices to try and find it. After running this tool many times over though it seems that my initial idea of just using 4000 would work since they all had that same device key anyway (thanks to all being built in the same way). Now according to the VMware API documentation on this function nearly all of those parameters you see up there are optional and earlier revisions of the code included only enough to change the DeviceName to Production without the API throwing an error at me. Frustrated I added in all the required parameters only to be greeted by the dreaded #2 NIC upon reboot.
It wasn’t going well for me, I can tell you that.
After digging around in the API documentation for hours and fruitlessly searching the forums for someone who had had the same issue as me I went back to tweaking the code to see what I could come up with. I was basically passing all the information that I could back to it but the problem still persisted with certain virtual machines. It then occurred to me that I could in fact pass the network card back as a parameter and then only change the parts I wanted to. Additionally I found out where to get the current ChangeVersion of the VM’s configuration and when both of these combined I was able to change the network VLAN successfully without generating another NIC. The resultant code is below.
VirtualVmxnet3 Card = (VirtualVmxnet3)NetworkCard;
VirtualMachineConfigSpec Spec = new VirtualMachineConfigSpec();
Spec.DeviceChange = new VirtualDeviceConfigSpec[1];
Spec.ChangeVersion = Guest.Config.ChangeVersion;
Spec.DeviceChange[0] = new VirtualDeviceConfigSpec();
Spec.DeviceChange[0].Operation = VirtualDeviceConfigSpecOperation.edit;
Spec.DeviceChange[0].Device = Card;
((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).DeviceName = "Production";
Guest.ReconfigVM_Task(Spec);
What gets me about this whole thing is that the VMware API says that all the other parameters are optional when its clear that there’s some unexpected behavior when they’re not supplied. Strange thing is if you check the network cards right after making this change they will appear to be fine, its only after reboot (and only on Windows hosts, I haven’t tested Linux) that these issues occur. Whether this is a fault of VMware, Microsoft or somewhere between the keyboard and chair is an exercise I’ll leave up to the reader but it does feel like there’s an issue with the VIM API. I’ll be bringing this up with our Technical Account Manager at our next meeting and I’ll post an update should I find anything out.
