VSAN 6.2 hybrid disk group performance degradation (2146267)

Symptoms

• After upgrading a Virtual SAN (VSAN) environment with Hybrid disk groups to version 6.2, some virtual machines resident on VSAN datastores may exhibit poor disk IO performance compared to previous versions of VSAN.
  
  For example, Virtual Machine Read and/or Write IOs may have poor response time than on Virtual SAN 6.1 or Virtual SAN 6.0.

• A significantly lower than expected read cache hit ratio is observed on Virtual SAN caching tier.
• A higher percentage of IOPS may be observed on capacity tier disks (Magnetic disks) on Hybrid diskgroups when compared to VSAN 6.0 or VSAN 6.1 systems.

Purpose

The VSAN 6.2 performance issue with hybrid disk groups is resolved in VMware ESXi 6.0 Patch Release ESXi600-201608001.

A hybrid disk group comprises of one Solid State Disks (SSD) for the cache tier and one or more Magnetic HDDs (MDs) for the Capacity tier.

Cause

In VSAN 6.2, new data services are introduced. One of these data services is Deduplication and Compression. Deduplication and Compression are not supported for Hybrid VSAN configurations. 

This issue is caused by VSAN 6.2 performing low level scanning for unique blocks, which is related to deduplication, can still occur on VSAN hybrid disk groups. This causes performance deterioration on Hybrid Disk groups, as it has a significant read caching performance impact on the SSD cache tier of VSAN disk groups.

Resolution

This issue affects VSAN 6.2 Hybrid deployments only. This issue is NOT applicable to All Flash VSAN Clusters.

This issue is resolved in VMware ESXi 6.0 Patch Release ESXi600-201608001 available at VMware Patch Downloads. For more information, see:

• How to download patches in My VMware (1021623)
• VMware ESXi 6.0, Patch Release ESXi600-201608001 (2145663)

To work around this issue if you do not want to upgrade, VMware advises to turn off the dedup scanner option on each VSAN node contributing to a Virtual SAN Hybrid cluster.

Prerequisites:

• A VSAN node must be placed in maintenance mode (with the Ensure Accessibility option) and a restart is required.
• These commands must be run on each ESXi host in the VSAN cluster.
• These commands result in persistent changes and remain configured across reboots.

To disable the dedup scanner:

1. Connect to your ESXi VSAN cluster  node using the ESXi Shell.
2. Run this command to turn off the dedup scanner:
   
   esxcfg-advcfg -s 0 /LSOM/lsomComponentDedupScanType
   
3. Verify the new setting using this command:
   
   esxcfg-advcfg -g /LSOM/lsomComponentDedupScanType
   
   
   • For Hybrid VSAN deployments, the value of lsomComponentDedupScanType is 0 when disabled.
   • The default value for lsomComponentDedupScanType is 2.
     
4. To apply the new setting on a Virtual SAN Cluster, select the Ensure Accessibility option and place each host in maintenance mode.
   
5. Reboot each host or node in a rolling fashion to ensure Virtual VSAN objects remain available.
   
   Note: The requirement for a reboot is to ensure any active dedup scanning session of existing data will be terminated in a consistent manner across a given cluster.