Earlier this month I completely re-worked my homelab – with both storage and hardware upgrades as well as the underlying hypervisors. If you’re interested in the changes made, see this post here. So how has the experience with the new gear been? In no particular order, here’s my notes:
- Not having to burn 8+GB for a vCenter VM is nice – especially for a lab
- XenCenter console is far simpler and less of a resource hog
- The Nutanix Prism web interface is simple and effective
- The CPU performace of the E3-1200 series is not quite the same as the E5, though this is barely noticeable; the 1U fan spin-up is an issue – I’d like to get a different chassis to address this issue
- Storage performance is outstanding – far better than it was when using the SAN\NAS
- Power usage
- Removing 8x 15K SAS drives from the storage server dropped load on the UPS from around 43% to below 25% – guess I need to resize that next
- I can turn off 1 of the virtualization hosts if needed – this reduces power load (and heat) even further
One of the more important changes that needed to be addressed quickly was backups. I had previously been using Veeam free edition with a PowerShell scheduled task – it was a bit difficult to get working since FreeNAS\Samba doesn’t work very well with computer accounts or services that run as Network Service etc. But once working, it worked well and was one less thing to worry about. So now what? The VMs that I had been backing up are all now running on XenServer – so how to back them up?
Thanks to Jan Sipke’s blog post here, it only takes a few commands to snapshot then export a VM. One note on the vm-export command in XenServer is to use the compress=true argument to get the size down a bit an not write any whitespace (note this does use a bit of dom0 CPU). All in all, this simple script does the job simply – and when the destination is a CIFS SR – it will export .XVA files to be backed up via CrashPlan.
I also found it necessary to replace the non-enlightened NIC with an e1000 NIC as this works far better and with more guest OSs – the procedure used is here. Additionally, with a few *nix VMs, I needed to use the following to set the boot order:
xe vm-param-set uuid= xx HVM-boot-policy="BIOS order"
as XenServer sometimes forces boot device for non-Windows guests.
All of the “throw-away” lab virtual machines will be running on a single Nutanix CE host. Why? A few reasons:
- If I’m going to be using lots of RAM for a central control VM, it should at least be doing some compression, dedupe, storage optimization, and presenting a simple, non-flash, easy to use management interface
- I like the idea of aggregating all available space across multiple drives both SSD for performance and HDD for capacity – if I need more capacity, add a drive
A few notes after using it for a few weeks:
- Yes, acropolis is very much like KVM
- I like the fact that a VM will boot to PXE if nothing else is bootable – this was not always the case as it previously needed to be set in ACLI
The image service is simply amazing to use – it takes pretty much any format of disk and makes it usable. And you’ve natively got tools that allow conversion between formats:
qemu-img convert -O raw kvm-disk.img xenserver-disk.raw
Which I needed to use once to go between hypervisors.
I will likely make the following changes to the lab in the near future:
- The FreeNAS server is now a bit of overkill as it is only hosting SMB shares
- I will likely swap motherboards and use the E5 board for XenServer and the E3 board for storage
- As it is only hosting SMB shares, it will likely be converted to a Windows server
- The XenServer host will get moved into a 2U chassis with quieter fans and a smaller PSU
All in all, the performance for both hosts with local storage is leaps and bounds better than the 15K drives ever was, even with the SSDs for ZIL and L2ARC – but don’t get me wrong, FreeNAS has been rock solid otherwise, it’s just hard to beat local SSDs, especially NVMe ones.
And no, I don’t have all of the super amazing integration bits of the big VMW, XenServer offers more than enough, especially given my labs focus. And Nutanix CE is just plain slick – it’s great to use and there’s lots more integration coming between Acropolis and the Citrix stack.