I see a lot of posts on the FreeNAS forums asking about performance for VMware\Hyper-V or any other hypervisor using FreeNAS as the backing storage for virtual machines. Then I see the specs of the systems those people are looking to build or already have built and I frown a bit.
First, let me say that FreeNAS is more than capable of handling the task of storage for your home virtual lab – if you do it right. Second, let me say to use NFS instead of iSCSI….I know what you’re thinking: “But NFS uses async writes and will be slower and it’s not multipath…” This is not a valid argument – see below under ‘SSD’. Then realize that not having VMFS on top of ZFS is a good thing…especially when you can see and manipulate the file system from your storage server. Even more especially when you take a storage snapshot and simply copy your VM files somewhere without having to worry about mounts, etc.
Let’s talk about hardware for your server:
- Server class hardware: this is a big one. YES you want (need) ECC memory. And trust me, you want an IPMI port
- The more ECC RAM the better – a starting point is 16GB
- At least 2 Intel or Broadcom 1Gbps network adapters (or 10Gbps)
- If you plan on using CIFS at all, a higher clock speed processor is better
- Buy some SSDs for L2ARC and ZIL
- Buy an IBM M1015 HBA and flash it to IT mode
- Lastly, buy your HDDs***
Lets discuss each of these items.
Server class hardware
This is really more for ECC memory support, but IPMI and reliability are even more reason to buy a server motherboard.
ECC Memory, and lots of it
Yes ECC memory is required!! This is your VM data that you’re talking about here! Since all of the data coming in to the storage server is stored in RAM until the transaction group is committed, your memory needs to be capable of detecting and correcting bit errors. Do not cheap out on this (I’ll show you where to cheap out) Read more here: http://doc.freenas.org/index.php/Hardware_Recommendations
Intel or Broadcom NICs
The reason for Intel and Broadcom specifically has to do with driver compatibility and performance. Back to server hardware again, most server class dual (or quad) PCI NICs are Intel or Broadcom based and will work great…and most server class motherboards use these chipsets as well. For 10Gbps NICs, Chelsio is the preferred vendor. Further, if you plan on using multiple protocols (NFS and CIFS for example) you will want at least 1 NIC for each. Cheap buy: HP NC360T dual 1Gbps PCIe
Faster processor clock speed for CIFS
This one is simple: SAMBA is single threaded, so faster clock speed beats more cores in this case.
Buy some SSDs
You may have read that you only need a separate ZIL and\or a separate L2ARC under certain circumstances…virtualization is one of those circumstances. There are two things you need to know about the SSDs to buy:
- Buy SSDs with capacitors to prevent data loss in the event of power failure. This allows the SSD to clear its internal buffer before fully losing power
- For the ZIL you will want a mirror of SSDs. Why? to prevent data loss. That’s why. If you lose a single SSD\ZIL, you potentially lose all the data in one transaction group if a power failure occurs. This is just another way of protecting your data
- You do not need huge capacity – in fact, for the ZIL, you only need a few GB – as in 8GB or less
Buy an IBM M1015 HBA
These are by far the best HBA to use with FreeNAS – here’s a few reasons: They’re cheap; they can be easily flashed to IT mode which gets rid of the boot BIOS on the card and presents the disks directly to the OS without any software RAID getting in the way; 8 SAS\SATA ports; you can have multiple of these HBAs in a single system
***Here’s the most important: Buy the right hard drives
If you are going to virtualize anything DO NOT USE 7200 RPM DRIVES and expect really good performance. Get some 10K or even 15K spindles from ebay, put them in a mirror and call it a day. Find 4 or 6 300GB 15K SAS drives, put them in mirrored pairs, and you’ll have screaming fast storage. You will save space with compression, so 600-900GB should be MORE than enough space. You can use 7200 RPM drives for CIFS storage…but I’m tired of seeing people virtualize on 7200 RPM drives – you are nearly doubling the IOPS by using 15K SAS drives. Sure, they’re probably old, and have some use on them coming from ebay, but FreeNAS doesn’t trust your HDDs anyway, so I would not worry about it – they’re cheap and fast. Use your 2TB 7200 intellipower drives for storing your audio\video\photo libraries – NOT your virtual machines.
Other miscellaneous notes
- Don’t waste HDD space on the FreeNAS OS – use an internal USB thumbdrive (No EMC ‘vault’ drives)
- Use reverse fan-out cables from the SAS ports on the M1015 for sata drives\backplanes, otherwise use SAS cables
- Read the FreeNAS forums – there’s tons of good info there