FreeNAS – ESX and NFS; Synchronous Writes and the ZIL

I’ve been testing FreeNAS lately – connecting ESX hosts via NFS for virtual machine storage. Being POSIX compliant, ZFS must abide by any calls made with the o_sync flag set meaning essentially that all disk activity must be written to stable storage before success is returned. This includes most commonly databases, file server operations and most importantly NFS. This means that the ZIL (ZFS intent log) will be used as a special place on disk in the pool to temporarily write data to from RAM – in its default configuration the ZIL is comprised of blocks from your zpool.

There are 2 ways to improve performance for NFS workloads – especially intensive ones such as VMware:

  1. Use an SSD (or mirrored pair of SSDs – especially if you’re not at ZFS v28) as a dedicated log device
  2. Disable the ZIL *Not recommended – can cause data corruption

To prove that the ZIL is being used excessively, you can temporarily disable it to see if performance improves. See the screenshot below from the VCenter console – datastore write latency for my VMs was averaging 9-11ms which is not great. The arrow indicates the point at which the ZIL was disabled:

While it is not a good idea to disable the ZIL, it clearly shows that it is being strained. Once disabled, the new average latency was 0-1ms and the systems were much more responsive.

The command to disable the ZIL is:

 

This was done for testing\demonstration only. To re-enable the ZIL back to its default configuration run this command:

For the final FreeNAS, a pair of SSDs will be used for the ZIL.

Leave a Reply