Video Card Hardware - vSphere 4

The boss is out this week and I got all of my work done for the day so I thought I would throw up another post that I have been working on. Blogging, and most anything else, is more difficult with a newborn in your life but I wouldn't have it any other way.

I was recently messing with vSphere in my lab environment and tweaking hardware information to determine what was new with vSphere and hardware version 7. One thing that appears to be added is the ability to edit Video Card settings. I began by adjusting the video ram to 32mb and playing with the VM. Imagine my surprise when trying to vMotion the VM and I received the error:

A general system error occurred: Failed to write checkpoint data (offset 33558328, size 16384): Limit exceeded KB 1011971

This is apparently a result of the vRAM assigned is greater than 30MB. This can be reduced in vSphere by:

1. Shut down the VM
2. Right click on the VM in the inventory and select Edit Settings
3. Click on Video Card
4. Change value under "Enter total video RAM" to below 30MB or select "Reserve memory for the following selection" radio button.

Given this limitation I wondered what effect that may have if you have specific hardware needs for VDI or desktops in general.

Display Resolutions at 24bit color depth

Resolution	Video Ram (MB) 24bit color	Video Ram (MB) 32bit color	2 Display 24bit color (MB)	2 Display 32bit color (MB)
800x600	1.37	1.83	5.49	7.32
1024x768	2.25	3	9	12
1280x1024	3.75	5	15	20
1600x1200	5.49	7.32	21.97	29.3
1152x864	2.85	3.8	11.39	15.19
1280x800	2.93	3.91	11.72	15.63
1440x900	3.71	4.94	14.83	19.77
1680x1050	5.05	6.73	20.18	26.91
1920x1200	6.59	8.79	26.37	35.16

Fortunately the only scenario listed here that exceeds 30MB is when running 1920x1200 on two monitors. There are more configurations available but I thought this would be high on practical limit. I sincerely doubt there will be too many people screaming for dual monitors at this resolution.

On the other side of the coin, attempting to run 16bit on lower resolutions yields the error that "The video memory is limited between 1.17 MB and 128 MB. Select different number for the monitors, screen resolution, or color depth."

Running multiple Displays requires that the Virtual Machine Version be at version 7. You can do this by Right Clicking on the guest VM and selecting "Upgrade Virtual Hardware" You will receive the following warning. Keep in mind that ESX 3.5 does not support version 7 so make the upgrade wisely.

Running multiple displays can also force you to consume more video memory causing you to exceed the 30MB limit for vMotion.

I recently posed the question to a VMware sales engineer about what effect this might have on hosts when utilizing virtual desktops and needing that additional video memory resources. He assured me that he does not know of anyone reaching a level of contention but it is something they will be investigating further. I'm open to any additional information that someone may have on the subject.

Configuring Network with PowerCLI and .CSV File

A couple of weeks ago I was in need to configure several new ESX servers with multiple NICs, vSwitches, and VLANs. Since I'm not one to do the same thing over and over again I turned to scripting. To further complicate things I wanted to allow the same script to create vSwitches, configure vMotion switch, add port groups, and create NIC Teams all from a prepopulated .CSV file. I'm doing this primarily so that all configuration could be done before the server arrived and so that I could create "Switch Profiles" when I add new hosts to clusters.

In the end I developed a script that reads from a .CSV file and then creates the desired switching. The first column tells the script what type of addition it is and the script processes the information for that object accordingly. I've noticed a few others have been posting recently about scripts on setting up portgroups, etc. and thought it was time to no longer keep this to myself (i.e. finally found the time to post). I too would like to also credit Hal Rottenburg's book Managing VMware Infrastructure with Windows Powershell which was very helpful for this and many other scripting projects I will be posting soon. Mike Laverick's post linked below helped in configuring the vMotion switch.

I hope you find this helpful. It has been HUGE in my environment.

Download the files here:
ESX-ConfigureSwitchCSV-Generic.txt
esx_switching_generic.csv

Links where others discuss switch configuration with PowerCLI.
http://halr9000.com/article/828
http://www.holy-vm.com/2009/10/01/adding-port-groups-with-powercli/
http://www.rtfm-ed.co.uk/?p=1514



Text of Script w/out comments

$getinfo = Import-Csv "\\server\share\file.csv"
Connect-VIServer -Server vCenterServer

$getinfo | % {
$Type = $_.Type #!!!! Case Sensitive !!!!!!
$gethost = Get-VMHost -Name $_.HostName
$SwitchName = $_.SwitchName
$PortGroup = $_.PortGroupName
$Nic = $_.NIC
$VLAN = $_.VLAN
$IP = $_.IP
$Subnet = $_.Subnet
$kernelGW = $_.KernelGW

If ($Type -eq "Switch") {
$gethost | New-VirtualSwitch -Name $SwitchName -Nic $Nic
}
If ($Type -ne "vMotion") {
$getswitch = Get-VirtualSwitch -VMHost $gethost -Name $SwitchName
}
If ($Type -eq "Team"){
$getswitch | Set-VirtualSwitch -Nic $Nic
}
IF ($Type -eq "Portgroup") {
$getswitch | New-VirtualPortGroup $PortGroup -VLanId $VLAN
}
IF ($Type -eq "vMotion") {

$newvswitch = New-VirtualSwitch -VMHost $gethost -Name $SwitchName -Nic $Nic
$vmotion = New-VirtualPortGroup -VirtualSwitch $newvswitch -Name $PortGroup
New-VMHostNetworkAdapter -VMHost $gethost -PortGroup $PortGroup -VirtualSwitch $newvswitch -IP $IP -SubnetMask $subnet -VMotionEnabled: $true

$vmhostnetwork = get-vmhostnetwork $gethost
set-vmhostnetwork -network $vmhostnetwork -vmkernelgateway $kernelGW
}
}

USB in ESX - How?

Since we brought it up and since I feel like a slacker for not posting in several weeks I thought I'd share my experience on managing USB devices in an ESX environment. In my previous post I mentioned a posting about connecting USB devices to ESX hosts and the many different ways for doing that in both a temporary and permanent fashion. I've never had much need for temporary connection but for a rock solid permanent solution I highly recommend AnywhereUSB by Digi - website.

Priced at just a couple of hundred dollars you are able to assign one of these an IP (or DHCP which I don't recommend), install the supplied software on your guest VM, plug in your USB device and you're up and running. Ok, so in all honesty here are some gotchas and other things to keep in mind:

1. One AnywhereUSB device to one operating system at a time. You can mix and match between physical and virtual as long as they have the provided client and drivers but only one server can talk to one of these guys at a time. This also means that even though it has 5 USB ports in it you can not map them to individual servers. Hopefully someone from Digi will correct me on this but that's just the way it is.

2. Give it a static IP. It makes it so much easier to track it down and to keep track of which one is which. If you purchase multiples of these you had better label the unit as well. You'll thank me for this one.

3. Follow the instructions for installation. I know you're gitty to have this device but if you skip a step you'll be sorry. Refer to #5.

4. Just because you have a gigabit network, you're limited to the speed of the AnywhereUSB device. I vaguely recall seeing that you may get USB2 speeds of 400mbps but I have not seen that as the case in production. If you have a 100Mb network then you know what your cap is already regardless.

5. Make sure you load the Windows generic drivers on your guest VM. ESX does not present USB to the VM, and you're not adding it to your VM's settings unless you've been doing some unsupported hacking, so the generic drivers were not installed when your OS was installed. In order to do this you'll need to copy the usbd.sy_ file from the \I386 directory of the Windows CD to the \system32\drivers directory and CHANGE THE FILE EXTENSION to usbd.sys. Reboot and then install the software. If you miss this step you'll be sorry.

6. I found it quite refreshing the first time I installed one of these guys and had the presence of mind to try it out first on a test server a snapshot before taking the first step. Always a good practice when installing something new for the first time.

7. You have to reboot at least once to use an AnywhereUSB device. That should be all though. Refer to #5

If you've kept these things in mind and follow the instructions you'll have many happy years of giving your VMs access to critical USB devices without unsupported hacking or sacrificing VMotion.


Links
Original Post - VMETC.Com
http://vmetc.com/2009/03/11/connecting-a-usb-device-to-an-esx-hosted-vm/

Digi - AnywhereUSB Website
http://www.digi.com/pdf/wp_ESXServer_AnywhereUSB.pdf

USB in ESX - Why Not?

I was recently talking with a friend about the beauty of USB devices in Workstation and we quickly began talking about the difficulties of USB devices in ESX. This is a problem I had already tackled some years ago with AnywhereUSB but earlier this evening I found a post discussing USB connectivity to an ESX hosted VM and felt it time to talk about it with you.

http://vmetc.com/2009/03/11/connecting-a-usb-device-to-an-esx-hosted-vm/

In the post the author describes several means for connecting USB devices in both temporary and permanent scenarios. I personally have never had much need for one time connectivity but since many of our applications require USB dongles for their license managers it's safe to say that connecting through the host or through RDP just won't do as a permanent solution. This is especially true as virtualization continues to branch out into the desktop arena where USB devices are more than just a pretty face, they're often a non-negotiable requirement.

While the post was extremely helpful one of the items that I feel the author failed to discuss is why VMware would continue to not support USB connectivity to guest VMs from their ESX hosts. Why wouldn't they indeed?

My immediate belief is that like most things it is for our own protection. Just imagine how upset your management, or you if you are the management, would be if DRS did not properly balance your resources, HA did not restart your server upon a failure, or you were not able to put a machine into maintenance mode due to a USB device preventing Vmotion because of a device associated to a single host. I know I wouldn't want to tell the boss that the thousands of dollars I fought hard for to purchase these features were all done in because of a USB dongle. Note: I have not yet tested this scenario but it seems reasonable given the results I've seen from having VMDKs and ISOs on local storage.

The greatest strength of VMware over its competitors is that they control the hardware and control the drivers that the VMs see. They have worked hard on their hypervisor to make these features available and I'm pretty sure that they won't open their arms to USB devices on the host anytime soon. After all, there are clearly many different options out there that solve this issue. Where's the benefit in complicating their already growing datacenter OS further?

Death by Snapshot

One of the greatest features in VMware's many virtualization technologies is the ability to take snapshots of virtual machines while they are powered on. Put simply, VMware snapshots save the system state of the virtual machine providing a restore point. This technology is available and works very well on a variety of VMware platforms including Workstation, Server, and ESX. Unfortunately this feature carries it's own risk to the performance and stability of your environment. In this entry I will discuss the destructive event of a snapshot consuming remaining available space on a datastore.

Recovery Outline at End of post

How snapshots work

In preparing for this entry I came across an extremely helpful post by Eric Siebert where he very clearly explains the many aspects of snapshots and how they work. I'll summarize a few important points in this section but highly recommend reading his full post.

Snapshots can be initiated and managed by either the VI Client or through the command line and once initiated, the snapshot makes the base VMDK read only and writes all changes to a new file. Each new differential (delta) file begins at 16MB and grows at 16MB increments as changes are made to the VM but naturally can not exceed the size of the original base file. These files will continue to grow until the snapshot is reverted (the changes are applied back to the base VMDK) or disk space is depleted, which makes for a stressful afternoon if this happens.

Drive space is gone!

I have seen it happen several times. Someone, or something, initiates a snapshot in your ESX environment and forgets to remove that snapshot when they have completed their task. My personal experience has found that this typically becomes a problem when backup software uses snapshots and the snapshot isn't merged back. Snapshot usage in this manner is common for backing up VMs since it allows the full disk to be placed in a read-only state so that the copy can continue without interrupting the machine's ability to operate.

A live snapshot delta file for most VMs will not likely grow very quickly, except for servers with higher amounts of disk I/O such as Exchange, SQL, or File shares (especially when using Windows Volume Shadow Copy). If a growing snapshot file isn't discovered and either applied or reverted then the snapshot could consume the remaining available storage. Once this happens the VM with the applied snapshot can no longer write its changes to the delta file causing the server to stop. Additionally, any other VMs writing to the filled datastore will also be forced to shut down. Fortunately any virtual machines on the datastore without snapshots, or an active swap file, will continue to run.

Note: If you are dealing with a single snapshot then no additional space is required to commit that snapshot to the original VMDK file, but I personally feel better when I have some extra room to move.

Make Room

If you are like me and have storage claustrophobia you want to make some room on your datastore. If the VM is a critical server it would be advisable to move another VM to a different datastore so that the afflicted VM can be restarted quicker and reduce any risks from trying to move a VM with a snapshot. I've seen mixed information about migrating with snapshots and it's feasibility. I'll leave it to the reader in their own situation but my suggestion is to play it safe and move another system without running snapshots.

Storage Vmotion immediately comes to mind for this situation. Unfortunately as Chad Sakac clearly explains in his blog post, Storage Vmotion requires creating a snapshot in order to operate. Consequently, with no drive space, we're left with the horrid task of intentionally taking down a server to make room. In my situation we took down our intranet server since it was one of the smaller servers, would take the least amount of time to migrate, and would cause the least impact on employee productivity. The time to complete this task will depend on several factors, specifically in regards to the type and speed of storage that you are using. This process took about 15 minutes for us to migrate our server to a new datastore.

Apply Snapshots

Once drive space has been created you should be allowed to start up the VM and commit the snapshots. Don't forget to turn the migrated server back on!

I have yet to receive a consensus on whether applying the snapshot on a live machine is better than leaving it powered off, but if the server in question is a main production box then it may be worth giving it a shot. You can expect that the process will take longer on a live machine. It will certainly be performing better than it was a few minutes ago!

Applying the snapshots can be a long grueling ordeal depending on the amount of space you have consumed. Be patient and do not be surprised if you see your task timeout in the VI client. VirtualCenter will timeout any task at 15 minutes but your process will still be running. Check to see if your process is complete by keeping an eye on the datastore browser in the VI client. You will be looking to see that the delta files are no longer there and you will also note that there is storage available in your datastore again. You may need to hit refresh occasionally in order to witness the disappearing files.

Our environment provided us with over 70GB of snapshot files over the course of 3 days time which took approximately 90 minutes to apply. Eric Siebert speaks to this in the second part of his snapshot post where he states that "A 100 GB snapshot can take 3-6 hours to merge back into the original disk." Suffice it to say that the larger the snapshot the longer it will take, and the more storage "cushion" you have on the datastore, the greater your risk for a long wait.

Recovery

Once the snapshots have merged back into the original file you can get the VM back up and running (if you haven't done so already) and then Storage Vmotion the server you moved previously if that is available in your environment.

Prevention

Vmware does not provide any tools natively for monitoring active snapshots in your ESX environment. Third-party applications are available to help automate the process of finding these active snapshots. I have not personally used them yet but Jason Boche mentions a few of them in his blog where he briefly displays Xtravirt Snaphunter, RVTools, and hyper9.

I will probably get my hands on a couple of these in the coming weeks and will certainly provide some posts. If you are in a fix to get some monitoring on your snapshots it looks as though SnapHunter can notify you via email when you have snapshots or even commit them if you so choose.

If you want to go low tech and only manage a few machines, you can check for snapshots by looking in the VI client or keeping an eye out for delta files in the datastore browser. Be vigilant regardless of your method for tracking active snapshots. It certainly doesn’t look good to the bosses when your highly robust ESX environment fails your company, especially when it can be easily prevented.

Despite the agony that can be caused by an unchecked snapshot, Vmware's snapshot feature is a true saving grace for the administrator and should be used without too much trepidation. The ability to apply a patch, test a deployment, or change a configuration and then quickly revert the system is more than I'd be willing to give up. Just keep your eyes open to the snapshots that are out there and everything should run smoothly and optimally, which the bosses definitely appreciate.

Recovery Outline

1. Identify the server(s) affected and determine priority on bringing them back online.
2. Shutdown and cold migrate another virtual machine from the filled datastore to a new location. Not always necessary if you have only a single snapshot since applying a single snapshot requires no disk space.
3. Apply snapshots to the affected server. You may power on server if you prefer but this will have an adverse effect on performance and cause this step to take longer.
4. Be patient. A 100GB snapshot could take 3-6 hours to commit. VirtualCenter will timeout your task after 15 minutes so don't panic.
5. Monitor the Datastore Browser in the VI client and wait for the delta file(s) to disappear. You will likely need to refresh occasionally which may take a moment to process each time.
6. Once the snapshot is committed you can safely turn on the VM (if you haven't already) and hopefully breathe a sigh of relief.
7. Play it safe and set up a system of monitoring your VMs for active snapshots through either an automated software like SnapHunter from Xtravirt or simply monitoring for delta files in the datastore browser