Managing Storage Policies using PowerCLI

posted in: Uncategorized | 0

It has been a personal project due to a few people asking me in working out how to create and manipulate vSAN Storage Policies from the new PowerCLI commandlets introduced in PowerCLi 6.5, the tasks I will cover here will be

  1. Creating a Brand New Storage Policy
  2. Changing an existing storage policy

For the purpose of this excersise I am using PowerCLI 6.5 Release 1 build 4624819 I owuld recommend you use that build or newer as some of the commands do not work as intended on previous PowerCLI builds.


Creating a Storage Policy

Before we proceed with creating a storage policy, you need to ensure you are logged into your vCenter server via PowerCLI and get a list of capabilities from the storage provider, for this we run the command Get-SpbmCapability and filter out the VSAN Policy capabilities

C:\> Get-SpbmCapability VSAN*
Name                                     ValueCollectionType ValueType         AllowedValue
----                                     ------------------- ---------         ------------
VSAN.cacheReservation                                        System.Int32
VSAN.checksumDisabled                                        System.Boolean
VSAN.forceProvisioning                                       System.Boolean
VSAN.hostFailuresToTolerate                                  System.Int32
VSAN.iopsLimit                                               System.Int32
VSAN.proportionalCapacity                                    System.Int32
VSAN.replicaPreference                                       System.String
VSAN.stripeWidth                                             System.Int32

For the definitions of the values associated with each definition:-
System.Int32 = Numerical Value
System.Boolean = True or False
System.String = “RAID-1 (Mirroring) – Performance” or “RAID-5/6 (Erasure Coding) – Capacity”

So I want to create a Policy with the following Definitions

Policy Name: PowerCLI-RAID5
Policy Description: RAID5 Policy Created with PowerCLI
Failures to Tolerate: 1
Failures to Tolerate Method: RAID5
Stripe Width: 2
IOPS Limit: 1000

To create a policy with the above specification we need to run the following command:

New-SpbmStoragePolicy -Name PowerCLI-RAID5 -Description "RAID5 Policy Created with PowerCLI" `
-AnyOfRuleSets `
(New-SpbmRuleSet `
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.hostFailuresToTolerate" ) -Value 1),`
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.stripeWidth" ) -Value 2),`
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.replicaPreference" ) -Value "RAID-5/6 (Erasure Coding) - Capacity"),`
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.iopsLimit" ) -Value 1000)`

We can then check for our policy by running the following command:

C:\> Get-SpbmStoragePolicy -Name "PowerCLi-RAID5" |Select-Object -Property *

CreatedBy       : Temporary user handle
CreationTime    : 11/04/2017 13:42:36
Description     : RAID5 Policy Created with PowerCLI
LastUpdatedBy   : Temporary user handle
LastUpdatedTime : 11/04/2017 13:42:36
Version         : 0
PolicyCategory  : REQUIREMENT
AnyOfRuleSets   : {(VSAN.hostFailuresToTolerate=1) AND (VSAN.stripeWidth=2) AND (VSAN.replicaPreference=RAID-5/6 (Erasure Coding) - Capacity) AND (VSAN.iopsLimit=1000)}
CommonRule      : {}
Name            : PowerCLI-RAID5
Id              : 9f8f61f4-24db-4513-b48f-a48c584f0eac
Client          : VMware.VimAutomation.Storage.Impl.V1.StorageClientImpl


And from the UI We can see our storage policy:

That’s our policy created, now we’ll move onto changing an existing policy.


Changing an existing storage policy

The policy we created earlier, if we suddenly decide that we want RAID 6 rather than RAID 5, we have to change the Failure To Tolerate number to 2, and we will also want to change our policy name and description as it will no longer be RAID5.  In order to change a policy, we have to ensure we specify the other values for the other policy definitions, if we do not specify the policy definitions that we do not want to be changed, they will be removed from the storage policy, so first we run the command that tells us what our policy definition is:

C:\> Get-SpbmStoragePolicy -Name "PowerCLi-RAID5" |Select-Object -Property *

CreatedBy       : Temporary user handle
CreationTime    : 11/04/2017 13:42:36
Description     : RAID5 Policy Created with PowerCLI
LastUpdatedBy   : Temporary user handle
LastUpdatedTime : 11/04/2017 13:42:36
Version         : 0
PolicyCategory  : REQUIREMENT
AnyOfRuleSets   : {(VSAN.hostFailuresToTolerate=1) AND (VSAN.stripeWidth=2) AND (VSAN.replicaPreference=RAID-5/6 (Erasure Coding) - Capacity) AND (VSAN.iopsLimit=1000)}
CommonRule      : {}
Name            : PowerCLI-RAID5
Id              : 9f8f61f4-24db-4513-b48f-a48c584f0eac
Client          : VMware.VimAutomation.Storage.Impl.V1.StorageClientImpl

So we need to make the following changes to the policy

  • Change the Failure to Tolerate value from 1 to 2
  • Change the Name of the Policy from PowerCLI-RAID5 to PowerCLI-RAID6
  • Change the Description of the policy from “RAID5 Policy Created with PowerCLI” to “RAID6 Policy Changed with PowerCLI”

We run the following command which specified the new values, and also the values we do not want to change:

Set-SpbmStoragePolicy -StoragePolicy PowerCLI-RAID5 -Name PowerCLI-RAID6 -Description "RAID6 Policy Changed with PowerCLI" `
-AnyOfRuleSets `
(New-SpbmRuleSet `
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.hostFailuresToTolerate" ) -Value 2),`
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.stripeWidth" ) -Value 2),`
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.replicaPreference" ) -Value "RAID-5/6 (Erasure Coding) - Capacity"),`
(New-SpbmRule -Capability (Get-SpbmCapability -Name "VSAN.iopsLimit" ) -Value 1000)`

We can verify the policy has been changed by running the command with the new policy name:

C:\> Get-SpbmStoragePolicy -Name "PowerCLi-RAID6" |Select-Object -Property *

CreatedBy       : Temporary user handle
CreationTime    : 11/04/2017 13:42:36
Description     : RAID6 Policy Changed with PowerCLI
LastUpdatedBy   : Temporary user handle
LastUpdatedTime : 11/04/2017 14:03:51
Version         : 2
PolicyCategory  : REQUIREMENT
AnyOfRuleSets   : {(VSAN.hostFailuresToTolerate=2) AND (VSAN.stripeWidth=2) AND (VSAN.replicaPreference=RAID-5/6 (Erasure Coding) - Capacity) AND (VSAN.iopsLimit=1000)}
CommonRule      : {}
Name            : PowerCLI-RAID6
Id              : 9f8f61f4-24db-4513-b48f-a48c584f0eac
Client          : VMware.VimAutomation.Storage.Impl.V1.StorageClientImpl

And again in the UI we can see the changes made

That’s how you change an existing storage policy, I hope this blog helps you manage your storage policies via PowerCLI, thanks go out to Pushpesh in VMware for showing me the error of my ways when trying to figure this out myself.






Cache Sizing

posted in: Uncategorized | 0

Since vSAN was released in 2014 there has been a bit of confusion as to how much cache should be sized for the cluster, this article is intended to clear that up and provide direction for both Hybrid and All-Flash configurations.  The reason there are differences in the recommendations is primarily because in Hybrid the cache is a Read Cache as well as a Write Buffer and in All-Flash it’s just serving as a Write Buffer, so the sizing is not a “one size fits all”.

Capacity Definitions:

  • RAW Capacity – This is the amount of capacity the vSAN Datastore will provide
  • Usable Capacity – This is the amount of capacity that can be provided based on the FTT level specified in storage policies
  • Provisioned / Deployed Capacity – This is the amount of space taken by objects before FTT is taken into account
  • Consumed Capacity – This is the amount of space that has been consumed by objects taking into account FTT, for example a 100GB Object with FTT=1 will consume 200GB of storage space


Hybrid Cache Sizing
In Hybrid the recommendation has always been 10% of Usable Capacity or Deployed capacity, if we have a 3-Node vSAN cluster, each host has two disk groups and each disk group has 7×1.2TB 10K SAS Drives, that means each host has 16.8TB of RAW Capacity and our 3-Node cluster has 50.4TB of RAW Capacity, based on the following FTT Values in our storage policy this means our total Usable Capacity is:

FTT=0 – 50.4TB
FTT=1 – 25.2TB
FTT=2 – 16.8TB
FTT=3 – 12.6TB

Based on the 10% rule our cache requirements are as follows:

FTT=0 – 5.4TB which equates to 1.8TB Per node which equates to 900GB Per Disk Group
FTT=1 – 2.52TB which equates to 0.84TB Per node which equates to 420GB Per Disk Group
FTT=2 – 1.68TB which equates to 0.56TB Per node which equates to 280GB Per Disk Group
FTT=3 – 1.26TB which equates to 0.42TB Per node which equates to 210GB Per Disk Group

The above sizing is all well and good if you are only using a single FTT method, however vSAN allows you to define policies with different FTT levels which means you can have objects on vSAN that have varying levels of protection, this makes sizing using the above method all the more difficult.

The best way to size the cache in a Hybrid cluster is to base it on your deployed or provisioned capacity, for example in the above RAW capacity of 50.4TB you may choose to have the following as an example

10 Objects based on FTT=0 of 500GB which totals 5TB of Provisioned Capacity and 5TB of Consumed Capacity
10 Objects based on FTT=1 of 500GB which totals 5TB of Provisioned Capacity and 10TB of Consumed Capacty
10 Objects based on FTT=2 of 500GB which totals 5TB of Provisioned Capacity and 15TB of Consumed Capacity
10 Objects based on FTT=3 of 500GB which totals 5TB of Provisioned Capacity and 20TB of Consumed Capacity

If you total up the above, our Provisioned Capacity is 20TB but our Consumed Capacity is 50TB, based on the Provisioned Capacity of 20TB, 10% of this is 2TB which equates to 0.67TB Per Node, or 333GB per Disk Group, this is how your cache in Hybrid should be sized.

All-Flash Cache Sizing
All flash has a lot more factors to consider, Erasure Coding, Dedupe and Compression and the fact that the cache is purely a Write Buffer so we have to take into account write endurance so the usual 10% sizing does not apply here.  In reality the typical 70% Read / 30% Write workload means that a lot of the requests are coming from the Capacity Tier which in this case is flash based anyway, so this means that the cache layer can be much smaller than it would have been in Hybrid for the same RAW capacity, however there is the write endurance factor to take into account.  We all know there is a write buffer limit in vSAN but that does not mean you should limit the size of the SSD drives based on that, the main reason is to increase the endurance of the drive, vSAN will cycle through all the cells on the drive irrelevant of the Write Buffer Limit.  VMware recently published a new sizing guide for All-Flash which is shown below


There we have it!



Creating a vSAN Cluster without a vCenter Server

posted in: Uncategorized | 2

I have been asked many times about creating a 3-node vSAN cluster without a vCenter server, the main reason for doing this is that you need to place your vCenter server onto the vSAN datastore but have no where to host the vCenter server until doing so.  The many customers I have spoken to are not aware that they can do this from the command line very easily.  In order to do this you must have installed ESXi 6.0 U2 and enabled SSH access to the host, there are a few steps in order to do this

  1. Configure the vSAN VMKernel Interface
  2. Create the vSAN Cluster
  3. Add the other nodes to the cluster
  4. Claim the disks

Step 1 – Create the VMKernel interface
In order for vSAN to function you need to create a VMKernel Interface on each host, this requires other dependencies such as a vSwitch and a Port Group, so performing this on all three hosts is a must so lets do it in this order, firstly lets create our vSwitch, since vSwitch0 exists for the management network we’ll create a vSwitch1

esxcli network vswitch standard add -v vSwitch1

Once our vSwitch1 is created we then need to add the physical uplinks to our switch, to help identify which uplinks to use we run the following command

esxcli network nic list

This should return details on all the physical network cards on the host for example:

Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description
vmnic0  0000:01:00.0 ntg3        Up   1000Mbps   Full   44:a8:42:29:fe:98 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1  0000:01:00.1 ntg3        Up   1000Mbps   Full   44:a8:42:29:fe:99 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2  0000:02:00.0 ntg3        Down 0Mbps      Half   44:a8:42:29:fe:9a 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3  0000:02:00.1 ntg3        Down 0Mbps      Half   44:a8:42:29:fe:9b 1500   Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4  0000:82:00.0 ixgbe       Up   10000Mbps  Full   a0:36:9f:78:94:cc 1500   Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic5  0000:82:00.1 ixgbe       Up   10000Mbps  Full   a0:36:9f:78:94:ce 1500   Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic6  0000:04:00.0 ixgbe       Up   10000Mbps  Full   a0:36:9f:78:94:c4 1500   Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic7  0000:04:00.1 ixgbe       Up   10000Mbps  Full   a0:36:9f:78:94:c6 1500   Intel Corporation Ethernet Controller 10 Gigabit X540-AT2

For my cluster I am going to add vmnic5 to the vSwitch1 so for this I run the following command:

esxcli network vswitch standard uplink add -v vSwitch1 -u vmnic5

Now that we now have our uplink connected to vSwitch1 we need to configure a portGroup for vSAN, for this I am calling my portGroup name “vSAN”

esxcfg-vswitch -A vSAN vSwitch1

Now we need to create out VMKernel interface with an IP Address ( for Host 1), Subnet Mask and assign it to the “vSAN” portGroup

esxcfg-vmknic -a -i -n -p vSAN

We validate our VMKernel Interface by running the following command:

[root@se-emea-vsan01:~] esxcfg-vmknic -l
Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack
vmk0       Management Network                      IPv4                    44:a8:42:29:fe:98 1500    65535     true    STATIC              defaultTcpipStack
vmk1       vSAN                                    IPv4                  00:50:56:6a:5d:06 1500    65535     true    STATIC              defaultTcpipStack

In order to add the VMKernel interface to vSAN we need to run the following command:

esxcli vsan network ip add -i vmk1

Repeat the above steps on the two remaining hosts that you wish to participate in the cluster

Step 2 – Creating the cluster
Once we have all the VMKernel interfaces configured on all hosts, we now need to create a vSAN Cluster on the first host, to do this we run the following command

esxcli vsan cluster new

Once completed we can get our vSAN Cluster UUID by running the following command:

[root@se-emea-vsan01:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2016-11-21T15:17:57Z
   Local Node UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
   Local Node Type: NORMAL
   Local Node State: MASTER
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 582a2bba-0fd8-b45a-7460-a0369f749a0c
   Sub-Cluster Backup UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
   Sub-Cluster UUID: 52bca225-0520-fd68-46c4-5e7edca5dfbd
   Sub-Cluster Membership Entry Revision: 6
   Sub-Cluster Member Count: 1
   Sub-Cluster Member UUIDs: 582a29ea-cbfc-195e-f794-a0369f7894c4
   Sub-Cluster Membership UUID: d2dd2c58-da70-bbb9-9e1a-a0369f749a0c

Step 3 – Adding the other nodes to the cluster
From the remaining hosts run the following command adding them to the newly created cluster

esxcli vsan cluster join -u 52bca225-0520-fd68-46c4-5e7edca5dfbd

You can verify that the nodes have successfully joined the cluster by running the same command we ran earlier noting that the Sub-Cluster Member Count has increased to 3 and it also shows the other sub cluster UUID Members:

[root@se-emea-vsan01:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2016-11-21T15:17:57Z
   Local Node UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
   Local Node Type: NORMAL
   Local Node State: MASTER
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: 582a2bba-0fd8-b45a-7460-a0369f749a0c
   Sub-Cluster Backup UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
   Sub-Cluster UUID: 52bca225-0520-fd68-46c4-5e7edca5dfbd
   Sub-Cluster Membership Entry Revision: 6
   Sub-Cluster Member Count: 3
   Sub-Cluster Member UUIDs: 582a29ea-cbfc-195e-f794-a0369f7894c4, 582a2bf8-4e36-abbf-5318-a0369f7894d4, 582a2c3b-d104-b96d-d089-a0369f78946c
   Sub-Cluster Membership UUID: d2dd2c58-da70-bbb9-9e1a-a0369f749a0c

Step 4 – Claim Disks
Our cluster is now created and we need to claim the disks in each node to be used by vSAN, in order to do this we first of all need to identify which disks are to be used by vSAN as a Cache Disk and as Capacity Disks, and obviously the number of disk groups, to show the disk information for the disks in the host run the following command:

esxcli storage core device list

This will produce an output similar to the following where we can identify the NAA ID for each device:

   Display Name: TOSHIBA Serial Attached SCSI Disk (naa.500003965c8a48a4)
   Has Settable Display Name: true
   Size: 381554
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.500003965c8a48a4
   Vendor: TOSHIBA
   Model: PX02SMF040
   Revision: A3AF
   SCSI Level: 6
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: unknown
   Other UIDs: vml.0200000000500003965c8a48a450583032534d
   Is Shared Clusterwide: false
   Is Local SAS Device: true
   Is SAS: true
   Is USB: false
   Is Boot USB Device: false
   Is Boot Device: false
   Device Max Queue Depth: 64
   No of outstanding IOs with competing worlds: 32
   Drive Type: physical
   RAID Level: NA
   Number of Physical Drives: 1
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   Emulated DIX/DIF Enabled: false

In my setup I want to create two disk groups per host consisting of 4 capacity devices plus my cache so to create one disk group I run the following command:

esxcli vsan storage add -s <naa for cache disk> -d <naa for capacity disk 1> -d <naa for capacity disk 2> -d <naa for capacity disk 3> -d <naa for capacity disk 4>

Once you have performed the above on each of your hosts, your vSAN cluster is deployed with storage and you can now deploy your vCenter appliance onto the vSAN datastore where then you can manage your vSAN License, Storage Policies, switch on vSAN Services such as iSCSI, health service and performance services as well as start to deploy virtual machines

Configuring iSCSI on vSAN 6.5

posted in: Uncategorized | 8

One of the new features of vSAN 6.5 is iSCSI access, so before I talk about how to configure iSCSI on vSAN let me just cover what is meant by iSCSI on vSAN to avoid any confusion 🙂

As we know, vSAN is an object based storage system, everything that exists on vSAN is an object that has a storage policy applied to is via Storage Policy Based Management, in vSAN 6.2 the vSAN performance metrics are stored on vSAN as an object also, so vSAN is already gearing up for other types of objects and not just a file system for running virtual machines.  The iSCSI side of vSAN is no different, the iSCSI Target has a home folder object just like a virtual machine, and each iSCSI LUN created has it’s own object, just like a VMDK does for a virtual machine.  The main use cases for vSAN are as follows:

  • Physical Workloads
  • In virtual machine Software iSCSI
  • Supports MPIO
  • Microsoft Cluster Service (MSCS) and Windows Failover Cluster Instance (FCI) (as at the time of writing this, the MSCS/FCI will be supported in a future release)

VMware does not allow the presenting of iSCSI LUNs on vSAN to other vSphere clusters, when I sit back and think about that statement I can see a few reasons for this

  • There is no SATP in ESXi for vSAN iSCSI
  • You lose the whole functionality Storage Policy Based Management for your virtual machines
  • You end up defeating the object of vSAN and start placing all of your Virtual Machines back onto LUNs again


So how do we configure iSCSI on vSAN?  Well just like the whole simplicity around vSAN, iSCSI is no different and is extrmely simple to set up, for the purpose of the test I used a Windows 2012 Server Virtual Machine and the iSCSI Initiator Service in Windows

Step 1 – Create an iSCSI VMKernel Interface
This interface will be used for iSCSI communication to allow the iSCSI traffic to flow over to our Physical or even Virtual Machines


Step 2 – Enable the iSCSI Service on vSAN


Suring the enabling you are presented with various options:

  • VMKernel Interface
  • TCP Port Number
  • Authentication Method (CHAP and Mutual CHAP suupported)
  • The Storage Policy to be applied to the Home Object for the iSCSI Target



Step 4 – Create an iSCSI LUN
For this I am creating an iSCSI LUN with a storage policy I have specified where my iSCSI LUNs are 100% Space reserved as per my Storage Policy, when creating the LUN it gives you an example on the right hand side of what this looks like from a vSAN Perspective, since I am using RAID1 – FTT=1, my 40GB LUN Object will consume 80GB of space and will be 80GB Reserved as per the policy


After this my display shows the LUN as being created and storage policy compliant


Step 5 – Add an Initiator or Initiator Group to the LUN Object
Before the Virtual Machine can see the LUN, I need to add the Initiator for the VM into the Allowed Initiators, I copied the initiator name from my virtual machine and pasted it in here


And the display shows this as now being added



From a vSAN perspective this is the iSCSI configuration completed, so now I need to go to my Windows virtual machine and configure the iSCSI Initiator side of things, so the first thing I need to do is enable MPIO for iSCSI, this is listed in Windows 2012 under Administrative Tools, ensure the “Add support for iSCSI devices” is selected


Once I have MPIO configured for iSCSI Devices it is time to add my iSCSI Target, for this you need to go into Adminisrative Tools and select the iSCSI Initiator service, once in there go to the “Discovery Tab” and enter the IP Address of your iSCSI Target, in my setup the iSCSI Target was being hosted by host 2 and for this the iSCSI IP is



Configure any advanced settings for example if you are using CHAP/Mutual CHAP, then click on the “Targets” tab and you will see your Target IQN as “Inactive”, click on Connect


In here select “Enable Multi Path”


Once connected I can now see my 40GB LUN in Device Manager and also in Disk Management, in Device Manager it is listed as “VMware Virtual SAN Multi-Path Disk Device”…..yes I know it’s no longer called Virtual SAN




That’s your iSCSI Configured on vSAN and also connected the LUN to a Windows 2012 Server, like I said, very easy to do

Virtual SAN – Easy to Use Proactive Tests

posted in: Uncategorized | 6

Since VMware introduced Virtual SAN back in March 2014 there have been a number of enterprise class features and enhancements, one of which was the Health UI, incorporated into the Health UI is a section called Proactive Tests which can be used to perform functionality testing and performance testing, there are three areas for the Proactive Tests they are:

  • VM Creation Test
  • Multicast Performance Test
  • Storage Performance Test
Proactive Tests
Proactive Tests Screenshot

The tests themselves are a good way to test that your cluster configuration is correct and complete, so what does each one of the tests do?

VM Creation Proactive Test

The VM Creation test is a simple deployment of a small sized virtual machine on each host within the cluster you have created, the test is designed to validate that:

  • Most aspects of the Virtual SAN configuration are completed
  • The whole management stack and Virtual SAN network is correctly defined and configured

The results of the test is passed through to the UI to help identify potential mis-configuration issues if the test fails to complete, in order for the result to be passed through as a pass, all hosts must return a success.

Multicast Performance Proactive Test

We all know that part of the networking requirements for Virtual SAN is Multicast, it is required from a cluster management perspective and not from a data flow perspective, it is pretty difficult if you are not in charge of your network infrastructure to perform a Multicast test on the network to make sure all is correct.  This test will actually perform a multicast test between all hosts and measure the speed, there are three levels of status with the multicast test and they are:

  • Failed – One or more hosts produced a multicast performance test of 20MB/Sec or below
  • Warning – One or more hosts produced a multicast test result of More than 20MB/Sec but less than 50MB/Sec
  • Passed – All hosts produced a multicast performance test result exceeding 50MB/Sec

Like the VM Creation test, the results are passed through to the UI depending on the above success criteria.  This allows you to check that your network supports multicast as well as the performance of multicast on the physical network.

Storage Performance Proactive Test

I personally like playing around with this one in my all-flash lab with different RAID levels and different performance profiles, I also use this a lot with customers just after they have set up their environment for a proof of concept, it allows you to get a baseline of performance for the cluster and troubleshoot any mis-configuration prior to running the proof of concept that could affect the performance.

Within this test, you can select the duration of the test, the storage I/O profile as well as the policy that is to be used for the storage performance test which is a huge advantage to be able to test different storage policies and the performance that they will give based on the definitions you specify in each policy

VSAN Proactive Storage Text

So apart from being able to choose how long to run the test for and the storage policy used for the test, what about the workload itself?  Well in the workload there are a number of tests that you can perform, there are enough tests to be able to generate a workload to simulate most real world workloads out there today, under the tests I have performed, it is pretty clear that the Storage Performance test here is pretty I/O intensive and after running the tests a few times it was clear that the tests also use 4K block sizes.  So what are the options in the Workload types?

Basic Tests

  • Low Stress Test – This workload will deploy a single disk object per host in the cluster and perform just a generic low utilization stress test on all the disk objects at the same time, this is one of the only test that utilizes a single object per host for testing and is not very I/O intensive
  • Basic Sanity Test, focus on Flash cache layer – Just like the Low Stress Test this will also deploy a single object per host and perform a sanity test which will as it states, focus on the cahe layer of the Virtual SAN storage, this test is more suited to Hybrid where there is a read cache tier

Intensive Tests

Apart from the two tests above, all the rest of the tests are designed to be I/O intensive and will show what capabilities your Virtual SAN cluster will deliver, each of the workloads below will deploy mutiple objects per host within the cluster and perform the test on them all at the same time, so be warned:

  • Stress Test – Like the Low Stress Test, this test will perform the same test but with 20 objects per host as opposed to a single object per host in the Low Stress Test and will also use a I/O block size of 8K
  • Performance characterization – 100% Read, optimal RC usage – This test is designed to test the performance of the read cache in a Hybrid cluster
  • Performance characterization – 100% Write, optimal WB usage – This test is deigned to test the write buffer in both Hybrid and All-Flash clusters
  • Performance characterization – 100% Read, optimal RC usage after warmup – This test will perform the same as the Optimal RC usage test, however it will not perform the test until the cache has been warmed up first, this will allow Virtual SAN to see what blocks are regularly being accessed in order to cache them
  • Performance characterization – 70/30 read/write mix, realistic, optimal flash cache usage – For most workloads this will be the test most commonly used
  • Performance characterization – 70/30 read/write mix, realistic, High I/O Size, optimal flash cache usage – This test uses a 64K block size for I/O testing
  • Performance characterization – 100% read, Low RC hit rate / All-Flash Demo – This will show the real Read performance of your All-Flash environment very well, I have used this test many times in my All-Flash environment
  • Performance characterization – 100% streaming reads – A test that would be the equivalent of streaming content from Virtual SAN via multiple sessions
  • Performance characterization – 100% streaming writes – A test that would be the equivalent of multiple video surveillance cameras streaming data to Virtual SAN

After the test has completed, the results will be displayed in the window below the test options like the following, as you can work out from my results below, the test I performed was for the High I/O Size as the MB/s divided by IOPS gives a block size of around 64K

VSAN Storage Performance Results

If we look at out performance charts for the same period of time we can see that the IOPS was over 88,000 for the cluster at 64K Block Size and a throughput of 5.4GB/second (43.2Gbps)

VSAN 64K IOPS Graphs

If we do the same for the All-Flash Demo these are the results based on the profile “Performance characterization – 100% read, Low RC hit rate / All-Flash Demo”

VSAN Storage Performance Results-AF

Again if we look at out performance charts for the same period of time we can see that the IOPS was over half a million for the cluster at 4K Block Size


Customers always ask me how realistic these tests are, I have done some comparisons with HCIBench also, and so have some of my customers, and HCIBench produces the same results, so as you can see the Virtual SAN Proactive Tests can save you a lot of time and effort with regards to things like storage performance testing, and remember they are very I/O intensive, coupled with the new performance monitoring charts in Virtual SAN, you have more tools at your fingertips all within a single user interface.

Enjoy 🙂







Drivers and Firmware Requirements

posted in: Uncategorized | 0

Out of all the conversations I have surrounding Virtual SAN with Customers, Partners and VMware folks, there is one topic that always surfaces and that topic is with regards to what drivers and firmware to use, is it the minimum version on the compatibility guide or what?  As we know Virtual SAN has a specific HCL when it comes down to three components

  • Disk Controller (VSAN RAID0 and Passthrough)
  • Solid State Drives
  • Magnetic Disks

So where does the confusion come from?

Historically, the compatibility guide for ESXi has always been supported as what was certified by the vendor, and anything newer than that is perfectly acceptable, but in the case of Virtual SAN this all changed, especially for the disk controller, and this is due to the whole interaction between Virtual SAN and the controller that the slightest change in firmware could result in a behavioural change on the controller which could have negative consequences on the Virtual SAN environment.  With Virtual SAN the hardware vendors will certify specific combinations of drivers and firmware, for example here

Virtual SAN Drivers and Firmware

You will see that the controller listed here for ESXi 5.5U3 (Virtual SAN 5.5) has two different drivers and associated firmware versions, this is the explicit combination that is required, so for example it would not be supported to use the driver version highlighted in red megaraid_perc9 version 6.901.55.00.1vmw with the firmware version highlighted in blue and vice versa, this is the exact firmware combination that must be used.  So if you update the firmware of the controller, you need to also ensure you apply the correct driver version also.

So what about newer versions you might ask?  Sometimes vendors will release newer Drivers and Firmware to provide fixes to known issues or enhancements to the controller, until that newer driver/firmware has been certified for Virtual SAN you should not upgrade to them, the chances are that it is in the process of being certified but until you see it on the Virtual SAN Compatibility Guide do not upgrade.  Stay Safe…..Stay Supported!

With regards to the Solid State Drives and Magentic disks, on the Virtual SAN Compatibility Guide you will see a firmware version published (Not in all cases for Magnetic disks), like the traditional ESXi Compatibility Guide, this is the minimum firmware supported, so as vendors produce newer firmware for them, you can update them as you wish, just remember to be on or above the mimumum published on the Virtual SAN Compatibility Guide

So just remember that:

  • Disk Controller – The exact driver / firmware combination on the Virtual SAN Compatibility Guide
  • Solid State Disks and Magnetic Disks – The firmware published on the Virtual SAN Compatibility Guide is the minimum version supported

Virtual SAN Compatibility Guide – Component Based

Configuring the Dell PERC H730 Controller for Passthrough and RAID

posted in: Uncategorized | 0

Sometimes during a deployment you might not be able to or want to install ESXi to the embedded SD Card controller on the Dell PowerEdge servers, for example if your memory configuration exceeds 512GB of RAM then installing ESXi to an SD Card would not be supported, or you may want to have a locally defined Scratch Partition on a local VMFS volume rather than using a remote Syslog or NFS Share, so what are the options?

Well you could boot ESXi from a single disk, the downside to that is if the disk fails then you are in a host down situation, the other option is to use two disks in a RAID1 mirror and use that for the ESXi installation.  With servers such as the R730XD you can utilise all 24 drive slots in the front of the server for Virtual SAN, and then two rear 2.5 Inch slots for your RAID 1 Mirror drives.  In my lab environment I have a similar config, I have a bunch of SSD drives used by Virtual SAN and then two 300GB 10K SAS Drives in a RAID 1 Mirror for my ESXi install and local VMFS Volume.  So how do we configure the H730 controller to support both Passthrough Disks and RAID disks?

Please note: For production environments this type of configuration is not supported

The first thing we need to do is ensure that the controller itself is configured for RAID Mode, for those that do not know, the H730 controller can be configured in either RAID Mode or HBA Mode.  RAID Mode allows you to create RAID Virtual Disks to be handled by the Logical Disk Controller as part of a RAID Volume, it also allows you to configure physical disks as NON-RAID disks, HBA Mode allows you to configure disks as NON-RAID capable only, in order to set the controller as RAID Mode we need to enter the System Setup Screen, to do this reboot the host and press F2 to enter the System Setup Main Menu, once there we need to enter the Device Settings option



This section of the System Setup will allow you to change the settings for any devices connected in the system, for the purpose of this article we will be focusing on the PERC H730 controller, in my servers I have two controllers, so I am selecting the one that is second in the list labelled RAID Controller in Slot 3: Dell PERC <PERC H730P Adapter> Configuration Utility



Once in the Device settings we need to chose the option for Controller Management and at the way to the bottom and choose the option for Advanced Controller Management, in there you will see an option for Switch to RAID Mode, when selected it will inform you that a reboot is required, do not reboot the host just yet.



Now we need to ensure that NON-RAID Disk Mode is enabled, for this, click on Back and enter the Advanced Controller Properties at the bottom of the screen and select the option to enable for Non RAID Disk Mode and hit Apply Changed



At this point you will need to make the disks you wish to use as a RAID1 as RAID Capable, you can do this under Configuration Management and choose the option to Convert to RAID Capable, from within here select the two disks you wish to use as your RAID 1 and click on OK, check the Confirm box and click on yes to confirm



Click on Back to go to the main menu and then re-select Configuration Management and chose the option to Create Virtual Disk and proceed to create your RAID 1 disk, it will only let you select disks that have been switched to RAID capable, after completion, check that all other disks are seen as NON-RAID Disks under Physical Disk Management, since we switched the controller from HBA Mode to RAID Mode all the other disks should still be tagged as NON-RAID

Now your PERC H730 controller is configured for both RAID and Passthrough and your RAID1 Virtual Disk can be used to install your ESXi, during the ESXi installation, make sure you install to the correct disk, and all your other disks are passthrough mode and can be consumed by Virtual SAN


Powering down a Virtual SAN Cluster

posted in: Uncategorized | 0

I speak with my customers and colleagues about this very topic, so I thought it was about time I wrote a post on how to power down a Virtual SAN cluster the correct way after all there may be countless reasons why you need to do this, from Scheduled Power Outages to Essential Building Maintenance.  There are a number of factors that have to be taken into account one major factor is when your Virtual Center server is actually running on the Virtual SAN cluster you wish to power down, in order to write this blog post I performed the actions on an 8-Node All-Flash cluster where the Virtual Center Appliance is also running within the Virtual SAN environment.

In order to perform the shutdown of the cluster you will need access to

  1. The vCenter Web Client
  2. Access to the vCenter
  3. SSH Access to the hosts, or access to the VMware Host Client

Step 1 – Powering off the Virtual Machines

Powering off the virtual machines can be done in multiple ways, the easiest way is through the WebUI, especially if you have a large number of virtual machines.  There are other ways that virtual machines can be powered off such as:

  • PowerCLI
  • PowerShell
  • vim-cmd directly on the ESXi hosts

VCVAShutdownscreenStep 2 – Powering down the Virtual Center Server

If your vCenter Server is running on your Virtual SAN datastore like my cluster then you will have to power down the vCenter Server after powering down the other virtual machines, if you are using the Windows vCenter Server you can simply RDP to the server and shut down the Windows Server itself, if you are like me and using the vCenter Appliance you can simply point a web browser to the vCenter IP or Fully Qualified Domain Name and Port 5480 and log in as the root user, from within there you will have the option to Shutdown the vCenter Appliance


Step 3 – Place the hosts into maintenance mode

Because our vCenter server is powered down, we won’t obviously be able to do this through the vCenter Web Client, so what are our easiest alternatives, well we can SSH into each host one at a time and run the command

# esxcli system maintenanceMode set -e -m noAction

The above command will place the host into maintenance mode and specifying “Do Nothing” as the response to the migration or ensuring that data will be accessible in the VSAN cluster, the reason behind this is that your virtual machines are powered off now anyway.

VSANHostClientOne of the other options and I would highly recommend this option is the VMware Host Client, this is basically a vib you install on each of your hosts, and then you point your web browser to the IP address or FQDN with a “/ui” at the end, this will bring you to a host based web client that is fantastic to use for managing an ESXi host outside of vCenter, from within this client you can place the host into maintenance mode from the Actions tab, and also shut the host down from the same tab.

One of the things I noticed when placing a host into maintenance mode here is that it did not present me with any options with regards to what to do with the data, the default behaviour here is “Ensure Accessibility”, due to the nature of this setting, this will prevent you putting all of your hosts into maintenance mode, so with the last hosts remaining, there are two choices

  • Log into the hosts via SSH and perform the command mentioned earlier
  • Shut the hosts down without entering maintenance mode

Since there are no running virtual machines, and the way Virtual SAN stores the disk objects there is no risk to data by doing either option, after all some power outages are unscheduled and Virtual SAN recovers from those seamlessly.


Step 4 – Power Down the ESXi hosts

Once all the hosts have been placed into maintenance mode we now need to power down any hosts that are still powered on from the previous step, this can be done without vCenter a number of ways:

  • Direct console access or via a remote management controller such as ILO/iDRAC/BMC
  • SSH access by running the command “dcui” and following the function keys to shutdown the system
  • From the ESXi command line by running the following command
    # esxcli system shutdown poweroff -r Scheduled
  • From the VMware Host Client as directed in the previous step


That’s it, your cluster is now correctly shut down, after your scheduled outage is over, the task of bringing up your cluster is straight forward

  1. Power up the hosts
  2. Use SSH or the VMware Host Client to exit maintenance mode
  3. Power up the vCenter server
  4. Power up the other virtual machines

Just a quick note, during the powering on of the ESXi hosts, Virtual SAN will check all the data objects, so whilst these checks are performed some objects or VMs may report as inaccessible from the vCenter UI, once Virtual SAN has performed its checks, the VMs will be readily available again.




Microsoft Windows Failover Clustering on Virtual SAN

posted in: Uncategorized | 6

In Virtual SAN 6.1 (vSphere 6.0U1) it was announced that there is full support for Oracle RAC, Microsoft Exchange DAG and Microsoft SQL AAG, I quite often get asked about the traditional form of Windows Clustering technology which is often referred to as Failover Cluster Instance (FCI).  In the days of Windows 2000/2003 this was called Microsoft Cluster Service (MSCS) and from Windows 2008 onwards it became known as Windows Failover Clustering, FailoverClustertypically this would consist of 1 Active node and 1 or more Passive nodes using a shared storage solution as per the image on the left.

When creating a failover cluster instance in a virtualised environment on vSphere the concept of shared storage is still a requirement, this could be achieved in one of two ways:

Shared Raw Device Mappings (RDM) – Where one or more physical LUNs are presented as a Raw Device to the cluster virtual machines either in Physical or Virtual Compatibility Mode, the Guest OS would write a file system directly to the LUNs.

Shared Virtual Machine Disks (VMDK) Where one or more VMDKs are presented to the cluster virtual machines.


With both MSCS and FCI all the virtual machines involved in the cluster would need to have their Virtual SCSI Adapter (where the shared RDMs or VMDKs are being attached) set to SCSI BUS Sharing in order to allow both VMs to access the same underlying disk, this would also allow the SCSI-2 (MSCS) and SCSI-3 (FCI) reservations to be placed on the disks by the clustering mechanism.

Now since Virtual SAN is not a distributed file system, and a disk object could be placed across multiple disks and hosts within the Virtual SAN cluster, the SCSI-2 and SCSI-3 reservation mechanism is non-existent, so the ability to use SCSI BUS Sharing will not work, remember each capacity disk in Virtual SAN is an individual contributor to the overall storage.  With Virtual SAN it is possible to create an MSCS/FCI by using the in guest Software iSCSI, to accomplish this we need the cluster nodes as well as an iSCSI target, the iSCSI Target will have VMDKs assigned to it like so:iSCSITarget

In the example to the right there are four VMDKs assigned, the first VMDK is the OS disk for the iSCSI Target and the other three disks are going to be used as the cluster shared disks, there is no need to set these disks to any type of SCSI BUS Sharing as the in guest iSCSI Target will manage these disks directly.  The Underlying VMDKs will still have a Virtual SAN Storage policy applied to them, so FTT=1 for example will create a mirror copy of the VMDKs somewhere else within the Virtual SAN Cluster.

The Cluster nodes themselves would be standard Windows 2003 / 2008 / 2012 virtual machines with the Software iSCSI Initiator enabled and used to access the LUNs being presented by the iSCSI Target, again since the in guest iSCSI will manage the LUNs directly there is no need to share VMDKs or RDMs using SCSI BUS Sharing on the virtual machine SCSI adapter, the Cluster nodes themselves can also reside on Virtual SAN, so you would typically end up with each of the cluster virtual machines accessing the shared storage via their in guest iSCSI initiator

MSCSLooksLikeSince each of the cluster nodes also reside on Virtual SAN, their OS disk can also have a storage policy defined, performing an MSCS / FCI this way does not infringe on any supportability issues as you are not performing anything you should not be from either a VMware or Microsoft perspective.  All the SCSI-2 / 3 Persistent reservations are handled within the iSCSI Target without hitting the Virtual SAN layer so providing the Guest OS for the iSCSI target supports this then there should be no issue and the MSCS/FCI should work perfectly, it is also worth noting that Fault Tolerance is supported on Virtual SAN 6.1, this could be used to protect the iSCSI Target VM compute against a host failure.

In my testing I used a Ubuntu Virtual Machine as well as a Windows Storage Server OS as the iSCSI Target virtual machine and Windows 2003/2008/2012 as the Clustering virtual machines hosting a number of cluster resources such as Sharepoint, SQL and Exchange

As newer clustering technologies come along, I see the requirement for a failover cluster instance to disappear, this is already happening with Exchange DAG and SQL AAG as the nodes replicate between themselves which mitigates the need for a shared disk between the nodes

One question I do get asked is how we support Oracle RAC on Virtual SAN as that uses shared disks also?  The answer to that is unlike MSCS/FCI which uses SCSI BUS Sharing, Oracle RAC uses the Multi Writer option in a virtual machine, Oracle RAC has a distributed write capability and it handles the simultaneous writes from different nodes internally to avoid data loss.  For further information on setting up Oracle RAC on Virtual SAN, please use KB Article 2121181




Virtual SAN ROBO

posted in: Uncategorized | 2

Along with the Stretched Cluster feature that arrived in Virtual SAN 6.1 (vSphere 6.0 U1) another feature that was delivered is the Remote Office Branch Office (ROBO) solution.  For many months customers who typically have a remote office branch office vSphere solution were not best pleased at the fact they needed a minimum of three Virtual SAN nodes per site.  With the ROBO solution the minimum number of hosts per site has changed and the licensing for ROBO is different than the conventional Virtual SAN solution as well.

ROBO solutions from VMware have been around for a while, there is a licensing model for vSphere that incorporates a ROBO solution, and in the days of VMware vCenter Storage Appliance (VSA) there was a ROBO solution for customers to have two ESXi hosts on site with a “Cluster” node as a thin client solution on site too, so technically you still needed three systems for VSA.  So how does Virtual SAN ROBO work?

Typical ROBO Setup:


The above image is typical of a Remote Office – Branch Office setup, all remote site vSphere clusters are managed centrally by a single Virtual Center server over standard WAN links, with the vSphere ROBO license pack you can run up to 25 virtual machines per site or even split the 25 Virtual Machines across multiple sites for example have 5 individual sites running 5 virtual machines under a single 25 virtual machine ROBO licence.  The Virtual SAN license for ROBO is exactly the same, it is purchased in packs of 25 virtual machines and those can be spread across multiple sites the same way that vSphere ROBO licence can be.

Virtual SAN ROBO Requirements:


Each remote site will host two ESXi servers running Virtual SAN allowing the RAID1 of disk objects to be fulfilled, in the centralized office there will be a witness host for each remote site, the witness is a virtualised ESXi host and performs the same functions as in the Virtual SAN Stretched Cluster. The storage requirements for the witness appliance also remain the same at 15MB per disk object, the maximum being 25 Virtual Machines having 60 disks each resulting in a maximum storage requirement for the witness appliance of around 23GB.

Unlike the stretched cluster the link between clusters and the witness is much lower, the diagram below shows the requirements:


As you can see from the above diagram, the witness has to reside on a different network to the hosts to prevent I/O traffic flowing across the WAN link and back to the other ESXi host, however the requirements for latency are much lower than the Stretched Cluster, this is mainly down to the maximum number of virtual machines per site being 25 versus the potential 1000’s for the Stretched Cluster.  You will also notice that 1Gb networking between the two hosts on the remote site is fully supported, again this is due to the maximum number of virtual machines that can be ran on the remote site.  It is still recommended to use multiple interfaces as an Active/Standby configuration in order to recover in the event of a network interface failure.

The main question I am asked is “Is it necessary to have a network switch on site”, the answer to this would be not necessarily, the VSAN VMKernel interfaces on the ESXi hosts have to be able to communicate with the Witness Appliance so they could be connected directly to a multiple port router that supports 1Gbps, in the event of a single port router a switch would be required, connecting a crossover cable between the ports on the hosts would not allow communication with the witness appliance and therefore it would not work.



Virtual SAN ROBO will be sold in packs of 25 licenses, this pack can be split across sites or used on a single site, this basically means you can run a maximum of 1 site with 25 virtual machines or up to 25 sites with a single virtual machine, it is a single license per site so there’s no going above 25 virtual machines per site with the ROBO License, you can split the license as any ways as you want providing you do not exceed the 25 virtual machines.

Over the past few days I have spoken with multiple customers about this solution, these customers come from a wide variety of sectors but the end goal remains the same, to reduce the hardware footprint at the remote sites and reduce management overhead, with Virtual SAN ROBO this is easily achieved the main points are:

  • No need to have a third host on site
  • No need for a shared storage array that is managed separately
  • Single CPU servers will also reduce the cost of hardware
  • Lower power consumption and hardware footprint
  • Single pane of glass to manage the remote cluster and storage








1 2