Yesterday I got an alert that a virtual machine needed consolidation. I opened my vSphere client and found it was in fact running on -000017.vmdk and looking at the directory I saw all of the snapshots. However, when I opened Snapshot Manager I saw no snapshots. My infrastructure uses Veeam B&R to run backups and traditionally if there is a snapshot on a VM the backup will fail. In my case, perhaps because VMware only kind of knew about it, it had been working. From the looks of it when Veeam talked with VMware’s API to consolidate it wasn’t happening properly. To delete all of the snapshots and consolidate down to one flat disk I found VMware KB article 1002310 that stated this could be resolved by the GUI or CLI by taking a new snapshot and deleting all snapshots. If that didn’t work it suggested that I quiesce the guest file system when I take a snapshot and try again. Well, this plan didn’t do the trick for me. I tried it with the VM off as suggested and that also did not work. Fortunately this was a VM that could be off without user impact. The next suggestion was cloning the VM and using the clone and decommissioning the VM with the snapshot problem. The last suggested “workaround” in the KB was to use vCenter Converter Standalone to essentially treat it like a P2V because it would only see the disks from the guest OS perspective. I wasn’t giddy about doing that. I decided to go with the clone method and it worked well for me. So, I thought I would share this KB with my blog crew. Be sure to consider data changes during the clone and take precautions such as disabling the network or shutting down the VM. For most production VMs this would probably need to be done during maintenance hours. I hope this is helpful for someone, but I imagine you may make it to the KB article just fine on your own. This client was using Veeam 6.5 and I upgraded them to Veeam 7.0 R2 after this issue and it worked properly last night.
The KB that shares all of this with details: Committing snapshots when there are no snapshot entries in the Snapshot Manager (1002310)
If you have a VMware VM that is running on a snapshot and the Snapshot Manager doesn’t see them you have a few ways to fix this.
- Create a new snapshot (check the Quiesce quest file system) and then Delete all snapshots. If this doesn’t work while the VM is running try with the VM turned off if possible.
- Create a clone of the VM and use the cloned VM and decommission the one with the snapshot problems.
- Run a P2V on the VM with the vCenter Standalone Converter and use the converted VM and decommission the one with snapshot issues
Other helpful KB articles when troubleshooting locks and snapshot issues
Other helpful recommendations for keeping a handle on snapshots
First, vCenter should be set to send you notifications of alerts. Secondly, I recommend adding your own custom alarm for a VM that is running on a snapshot too long. Snapshots are meant to be brief and as I mentioned above having a leftover snapshot can even cause backups to fail. I have heard of VMs running on hundreds of snapshots and you can imagine what a mess that would be to deal with. The maximum supported snapshots in a chain is 32, but it is recommended to not have more than 2-3 snapshots. Depending on the IO of your VM consolidations can be painful if you let it go to long. So, create an alarm to notify you when a VM is running on a large snapshot. This will notify you if you just forget to consolidate a snapshot. See this helpful KB on Configuring VMware vCenter Server to send alarms when virtual machines are running from snapshots (1018029). I recommend creating a rule at a high level to alert if a VM runs on a large snapshot for your environment so that you will be notified of that. This is tricky to determine the threshold, but considering backups happen on off-hours and IO should be relatively low I would start with a lower threshold. Every environment is different, but I will set mine as low as 1GB at times.
I was on vacation last week and received the great news that Veeam had delivered the patch for Veeam Backup and Replication version 7 that will extend support for VMware vSphere 5.5, Windows Server 2012 R2, and Windows 8.1. At least those are the ones that I am looking forward to. I want to give a big SHOUT OUT and THANK YOU to the Veeam Support Team for working tirelessly to deliver support for the latest platforms. I have to say that every time I call Veeam for support I get very informative and knowledgable support technicians who handle my incidents with great attention to detail and solutions. I love the follow-up emails to ensure that I am good to go. I always sell the support with Veeam Backup and Replication for my clients because they release patches like this one to keep my clients as up-to-date as they want to be and whenever I run into an issue I know I will have a good experience getting it resolved. So, thank you Veeam!
So, if you are running Veeam I highly recommend staying up-to-date to be ready for your clients infrastructure when it is time to update them to VMware vSphere 5.5, Windows Server 2012 R2, or Windows 8.1. First ensure you are on version 18.104.22.1680 or 22.214.171.1245 and then install the patch linked below to be on version 126.96.36.1994.
The Microsoft update cycle is really ramping up to annual releases of Windows and Windows Server. With that comes upgrades to other core infrastructure components such as for most of my clients VMware vSphere and Veeam Backup and Replication. As of today we begin ordering Windows Server 2012 R2 for clients production environments and that means they also need to upgrade their VMware vSphere infrastructure to 5.5. However, this cannot happen just yet and I will tell you why.
Veeam Backup and Replication version 7 has received its first patch, but take note it is not the compatibility patch we’ve been ready for. It is essentially a minor patch that address some of the issues with a new release.
Currently Windows Server 2012 R2, Windows 8.1, and vSphere 5.5 are not compatible with Veeam Backup and Replication v7 Patch 1. According to the data sheets below published on their web site the “compatibility” patch will be released in Q4 2013. On their web site it can be found to say late Q4 2013. I have heard some say it will be out toward the end of December. Today however I received a response on Twitter from @veeam where they said they will be delivering the update in November.
Here’s the data sheet on Veeam Backup and Replication 7 support for VMware vSphere 5.5.
Here’s the data sheet on Veeam Backup and Replication 7 support for Microsoft Windows Server 2012 R2 and Windows 8.1.
So, before you Veeam lovers go and roll out VMware vSphere 5.5, Microsoft Windows Server 2012 R2, and Microsoft Windows 8.1 in your production environments be sure to make sure the next “compatibility” patch is released first so you can protect those precious VMs with the best protection around.
I am getting my presentation finished up for my session at the IT Pro Camp event at Northwest Florida State College next weekend and I am pretty stoked.
I will be looking at a virtualization solution from a cost, HA/DR/BC, and management standpoint. I will be showcasing how innovations to Hyper-V with Hyper-V Server 2012 R2 are changing the game for the SMB. Looking at VMware ESXi 5.5 free hypervisor and Hyper-V Server 2012 R2 free offering to see what value you can get out of them as an SMB. I will have a live lab demonstration with a Hyper-V 2012 R2 host cluster and will go over so many exciting features.
I will be posting my slide deck, possibly videos from the presentation, and a PDF with the details after the presentation. So check back after the October 5th!
If you are in the Northwest Florida area I hope you can make it to this free event at IT Pro Camp on October 5th.
Well so far today at lunch I was able to get started with VSAN, but I still have lots to figure out it seems. Got a few hosts and vCenter setup. Got connected with the web client. Setup a network for VSAN. Turned off HA so I could turn on VSAN on the cluster? It is showing my SSD datastore as Non-SSD. Well, I will tackle this tonight or tomorrow. Excited either way.
At VMworld 2013 in some of the break-out sessions and booths presenters have been using the VMware Product Walkthrough demos that were created by VMware’s Technical Marketing team to demonstrate some new features with VMware vCloud Suite 5.5 series. It was announced today that they have been made available online for those walking to see it first hand. It is easy to do as it doesn’t require any registration or login.
You can walkthrough the new Virtual SAN feature, vSphere Data Protection, vSphere App HA, vCloud Director, vSphere Replication, and vSphere Flash Read Cache.
Anyhow, sometimes it helps to see an overview before you dive in and learn about all of the nuts and bolts behind the scene.
Here is the link to get started:
If you are like me and can’t make it to every tech gathering, but love to stay in the loop on cutting edge tech. Check out this VMworld session for an overview on the Virtual SAN (vSAN).
I wouldn’t rush this into production, but more and more of the datacenter is becoming software defined and will continue to lower TCO and ROI on your infrastructure. So, stay on it and move with the momentum.