Posts Tagged‘vsphere’

VMware VIM SDK Gotchas (or Ghost NICs, Why Do You Haunt Me So?).

I always tell people that on the surface VMware’s products are incredibly simple and easy to use and for the most part that’s true. Anyone who’s installed an operating system can easily get a vSphere server up and running in no time at all and have a couple virtual machines up not long after. Of course with any really easy to use product the surface usability comes from an underlying system that’s incredibly complex. Those daring readers who read my last post on modifying ESXi to grant shell access to non-root users got just a taste of how complicated things can be and as you dive deeper and deeper into VMware’s world the more complicated things become.

I had a rather peculiar issue come up with one of the tools that I had developed. This tool wasn’t anything horribly complicated, all it did was change the IP address of some Windows servers and their ESXi hosts whilst switching the network over from the build VLAN to their proper production one. For the most part the tool worked as advertised and never encountered any errors, on its side at least. However people were noticing something strange about the servers that were being configured using my tool, some were coming up with a “Local Area Network 2” and “vmxnet3 Ethernet Adapter #2” as their network connection. This was strange as I wasn’t adding in any new network cards anywhere and it wasn’t happening consistently. Frustrated I dove into my code looking for answers.

After a while I figured the only place that the error could be originating from was when I was changing the server over from the build VLAN to the production one. Here’s the code, which I got from performing the same action in the VIClient proxied through Onyx, that I used to make the change:

            NameValueCollection Filter = new NameValueCollection();
            Filter.Add("name", "^" + ServerName);
            VirtualMachine Guest = (VirtualMachine)Client.FindEntityView(typeof(VirtualMachine), null, Filter, null);
            VirtualMachineConfigInfo Info = Guest.Config;
            VirtualDevice NetworkCard = new VirtualDevice();
            int DeviceKey = 4000;
            foreach (VirtualDevice Device in Info.Hardware.Device)
                String Identifier = Device.ToString();
                if (Identifier == "VMware.Vim.VirtualVmxnet3")
                    DeviceKey = Device.Key;
                    NetworkCard = Device;
                    Console.WriteLine("INFO - Device key for network card found, ID: " + DeviceKey);
            VirtualVmxnet3 Card = (VirtualVmxnet3)NetworkCard;
            VirtualMachineConfigSpec Spec = new VirtualMachineConfigSpec();
            Spec.DeviceChange = new VirtualDeviceConfigSpec[1];
            Spec.DeviceChange[0] = new VirtualDeviceConfigSpec();
            Spec.DeviceChange[0].Operation = VirtualDeviceConfigSpecOperation.edit;
            Spec.DeviceChange[0].Device.Key = DeviceKey;
            Spec.DeviceChange[0].Device.DeviceInfo = new VMware.Vim.Description();
            Spec.DeviceChange[0].Device.DeviceInfo.Label = Card.DeviceInfo.Label;
            Spec.DeviceChange[0].Device.DeviceInfo.Summary = "Build";
            Spec.DeviceChange[0].Device.Backing = new VMware.Vim.VirtualEthernetCardNetworkBackingInfo();
            ((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).DeviceName = "Production";
            ((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).UseAutoDetect = false;
            ((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).InPassthroughMode = false;
            Spec.DeviceChange[0].Device.Connectable = new VMware.Vim.VirtualDeviceConnectInfo();
            Spec.DeviceChange[0].Device.Connectable.StartConnected = Card.Connectable.StartConnected;
            Spec.DeviceChange[0].Device.Connectable.AllowGuestControl = Card.Connectable.AllowGuestControl;
            Spec.DeviceChange[0].Device.Connectable.Connected = Card.Connectable.Connected;
            Spec.DeviceChange[0].Device.Connectable.Status = Card.Connectable.Status;
            Spec.DeviceChange[0].Device.ControllerKey = NetworkCard.ControllerKey;
            Spec.DeviceChange[0].Device.UnitNumber = NetworkCard.UnitNumber;
            ((VirtualVmxnet3)Spec.DeviceChange[0].Device).AddressType = Card.AddressType;
            ((VirtualVmxnet3)Spec.DeviceChange[0].Device).MacAddress = Card.MacAddress;
            ((VirtualVmxnet3)Spec.DeviceChange[0].Device).WakeOnLanEnabled = Card.WakeOnLanEnabled;

My first inclination was that I was getting the DeviceKey wrong which is why you see me iterating through all the devices to try and find it. After running this tool many times over though it seems that my initial idea of just using 4000 would work since they all had that same device key anyway (thanks to all being built in the same way). Now according to the VMware API documentation on this function nearly all of those parameters you see up there are optional and earlier revisions of the code included only enough to change the DeviceName to Production without the API throwing an error at me. Frustrated I added in all the required parameters only to be greeted by the dreaded #2 NIC upon reboot.

It wasn’t going well for me, I can tell you that.

After digging around in the API documentation for hours and fruitlessly searching the forums for someone who had had the same issue as me I went back to tweaking the code to see what I could come up with. I was basically passing all the information that I could back to it but the problem still persisted with certain virtual machines. It then occurred to me that I could in fact pass the network card back as a parameter and then only change the parts I wanted to. Additionally I found out where to get the current ChangeVersion of the VM’s configuration and when both of these combined I was able to change the network VLAN successfully without generating another NIC. The resultant code is below.

            VirtualVmxnet3 Card = (VirtualVmxnet3)NetworkCard;
            VirtualMachineConfigSpec Spec = new VirtualMachineConfigSpec();
            Spec.DeviceChange = new VirtualDeviceConfigSpec[1];
            Spec.ChangeVersion = Guest.Config.ChangeVersion;
            Spec.DeviceChange[0] = new VirtualDeviceConfigSpec();
            Spec.DeviceChange[0].Operation = VirtualDeviceConfigSpecOperation.edit;
            Spec.DeviceChange[0].Device = Card;
            ((VirtualEthernetCardNetworkBackingInfo)Spec.DeviceChange[0].Device.Backing).DeviceName = "Production";

What gets me about this whole thing is that the VMware API says that all the other parameters are optional when its clear that there’s some unexpected behavior when they’re not supplied. Strange thing is if you check the network cards right after making this change they will appear to be fine, its only after reboot (and only on Windows hosts, I haven’t tested Linux) that these issues occur. Whether this is a fault of VMware, Microsoft or somewhere between the keyboard and chair is an exercise I’ll leave up to the reader but it does feel like there’s an issue with the VIM API. I’ll be bringing this up with our Technical Account Manager at our next meeting and I’ll post an update should I find anything out.

Facts and Figures: ESX vs Hyper-V.

Virtualization, a word that started out as one of those industry buzz terms many years ago has managed to find its place in every single organisation I’ve had the pleasure of working for. It’s really quite a nifty technology, being able to run multiple virtual servers on a single physical box which allows you to better utilize your computing resources (and therefore dollars to). I was lucky enough to catch the virtualization bandwagon early on and managed to establish myself as a specialist in the field before most people had a clue how to utilize it properly. Primarily most of my experience was with VMware, since they were the only company doing it right for a very long time. Since then I’ve had a chance to sit down with Microsoft’s answer to ESX in the form of Hyper-V and the differences are quite significant. So in this battle royal-esque show down I’ll pit them against each other and we shall see who comes out the victor.

A great judge of how well a company can do something is how long they’ve been in business. VMware started out in 1998 and delivered their first virtualization product, VMware workstation, a year later in 1999. Microsoft delivered the first version of Hyper-V back in 2008, but we could safely say they were working on it since at least 2007. That gives them 11 and 2 years respectively in the virtualization game, so the advantage of time in the market is going to be handed to VMware.

VMware: 1, Microsoft: 0

One of the biggest considerations that most organisations have when looking to implement virtualization is the cost, as they’re usually trying to save money by implementing it. For the most part they will as long as the design is done properly and they’re not virtualizing for virtualization’s sake. However this is one of the glaring points that plagues VMware, it’s almost ludicrous prices. A single core standard edition license will run you about US$795 and the creme de’la creme costs a whopping US$3,995. That’s not to mention the cost of the central management server either, which will cost you another $4,995 plus the $1,095 required for one year of support. Hyper-V on the other hand is an additional US$28 on top of your existing license fee for Windows Server 2008, which you’re probably going to be buying anyway. The central management server costs about US$1150 which is a darn sight cheaper than VMware. The Hyper-V licensing model also has the advantage of usually being wrapped up with your licensing agreement with Microsoft, flying under the budget radar.

VMware: 1, Microsoft: 1

Now I’m going to get into the nitty gritty of these virtualization solutions: the architecture. Thanks to their marketing departments both of them give you a high level view of their respective virtualization systems.


(Click to enlarge)

Under the hood they’re very similar in the way they’ve designed their solutions and really there’s not much they could have done differently. The biggest difference that’s not clearly highlighted in these marketing speals is that the underlying technology for VMware is basically a modified Red Hat Enterprise Linux kernel that runs directly on the hardware. Hyper-V on the other hand requires that Windows Server 2008 be installed first and then the virtualization run on top of that. While the differences between these two are small, architecturally speaking, the main difference comes from the fact that for Hyper-V you’re running a full blown copy of the operating system in order to get the virtualization benefits. VMware avoids this by using a very slimmed down version of the Linux kernel (the footprint is about 32MB) meaning that more resources are available for the guests. Microsoft has tried to counter this somewhat by introducing a version of Windows Server 2008 called Server Core which removes quite a bit of the cruft, but the large memory footprint and installbase remains.

VMware: 2, Microsoft: 1

The next set of comparisons get a bit fuzzy, since they rely heavily on which version of ESX you end up buying from VMware. For instance if you buy the standard or advanced versions of ESX you’re basically buying Hyper-V at an inflated price, since the feature sets are basically the same. Up until the recent release of R2 for Hyper-V the advanced licensing for ESX would still bring out on top feature wise, still at a premium however. Since VMware has been in the virtualization game for so long they’re no longer focused on what most people would consider core functionality so the more advanced licenses are where they really begin to shine. Things like dynamic resource management and live storage migrations are just simply not available for Hyper-V and have been deal breakers for me in a couple situations. Additionally ESX is very agnostic when it comes to what you install on it, supporting basically every operating system and their respective versions out there. On a pure feature level VMware wins out, but not without incurring some hefty costs.

VMware: 3, Microsoft: 1

One consideration that most people leave out is how easily a virtualization solution will integrate into their environment. VMware is a completely different way of doing things and whilst vSphere is leaps and bounds easier to use than its predecessors it’s still a completely different world to your usual world of stumbling through menu options to fix a problem. Your decent ESX administrator will have to know his way around a Linux command line in order to get some of the more advanced things done (just try to install EMC SAN agents without it, I DARE YOU! :P) and there’s a fair share of problems that will require some obscure commands that can do a heck of a lot of damage if you don’t know what you’re doing. Granted there is quite a good community for all of VMware’s products so you’re not usually left in the dark with no one to hold your hand, but its still no substitute for an ESX administrator who’s dealt with his share of broken ESX environments before. Hyper-V on the other hand has that distinct Microsoft flavour to it and any Windows administrator will be able to administer and troubleshoot an installation without too much hassle. This also has the benefit of not attracting a premium when it comes to hiring in new talent, as us ESX know-it-alls love to charge a bit extra.

VMware: 3, Microsoft: 2

Even though the final score here would lead you to believe that VMware has come out victorious I’m still hesitant to give it the crown of virtualization king (and yes I know I’ve left out Xen but that’s with good reason, I’ve never used the damn thing). The cost of VMware is so massive that I find it hard to convince any shop who’s looking to virtualize to go with it. Sure the name carries a heck of a lot of weight and really if you’re paying the prices on the website you need to talk to a different VMware rep but the point remains that even a single site installation with the basic package will easily run up US$10,000 on top of your current Windows licensing costs. For anyone with a lot of disperate sites that require a virtualization solution you’re much better off with Hyper-V, although ESX really shines in large (100+ servers) data centers. I still recommend VMware to most people since they really are the best, despite their high costs. It would seem as always that you need to carefully analyze your situation and build a solution based on your requirements and not the marketing hyperbole.

Although if you use VMware you’re more likely to get a charming person like myself knocking at your door, and who doesn’t want that? 😉