Creating a vSAN Cluster without a vCenter Server

I have been asked many times about creating a 3-node vSAN cluster without a vCenter server, the main reason for doing this is that you need to place your vCenter server onto the vSAN datastore but have no where to host the vCenter server until doing so.? The many customers I have spoken to are not aware that they can do this from the command line very easily.? In order to do this you must have installed ESXi 6.0 U2 and enabled SSH access to the host, there are a few steps in order to do this

  1. Configure the vSAN VMKernel Interface
  2. Create the vSAN Cluster
  3. Add the other nodes to the cluster
  4. Claim the disks

Step 1 – Create the VMKernel interface
In order for vSAN to function you need to create a VMKernel Interface on each host, this requires other dependencies such as a vSwitch and a Port Group, so performing this on all three hosts is a must so lets do it in this order, firstly lets create our vSwitch, since vSwitch0 exists for the management network we’ll create a vSwitch1

esxcli network vswitch standard add -v vSwitch1

Once our vSwitch1 is created we then need to add the physical uplinks to our switch, to help identify which uplinks to use we run the following command

esxcli network nic list

This should return details on all the physical network cards on the host for example:

Name??? PCI????????? Driver????? Link Speed????? Duplex MAC Address?????? MTU??? Description
vmnic0? 0000:01:00.0 ntg3??????? Up?? 1000Mbps?? Full?? 44:a8:42:29:fe:98 1500?? Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic1? 0000:01:00.1 ntg3??????? Up?? 1000Mbps?? Full?? 44:a8:42:29:fe:99 1500?? Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic2? 0000:02:00.0 ntg3??????? Down 0Mbps????? Half?? 44:a8:42:29:fe:9a 1500?? Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic3? 0000:02:00.1 ntg3??????? Down 0Mbps????? Half?? 44:a8:42:29:fe:9b 1500?? Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
vmnic4? 0000:82:00.0 ixgbe?????? Up?? 10000Mbps? Full?? a0:36:9f:78:94:cc 1500?? Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic5? 0000:82:00.1 ixgbe?????? Up?? 10000Mbps? Full?? a0:36:9f:78:94:ce 1500?? Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic6? 0000:04:00.0 ixgbe?????? Up?? 10000Mbps? Full?? a0:36:9f:78:94:c4 1500?? Intel Corporation Ethernet Controller 10 Gigabit X540-AT2
vmnic7? 0000:04:00.1 ixgbe?????? Up?? 10000Mbps? Full?? a0:36:9f:78:94:c6 1500?? Intel Corporation Ethernet Controller 10 Gigabit X540-AT2

For my cluster I am going to add vmnic5 to the vSwitch1 so for this I run the following command:

esxcli network vswitch standard uplink add -v vSwitch1 -u vmnic5

Now that we now have our uplink connected to vSwitch1 we need to configure a portGroup for vSAN, for this I am calling my portGroup name “vSAN”

esxcfg-vswitch -A vSAN vSwitch1

Now we need to create out VMKernel interface with an IP Address (192.168.100.1 for Host 1), Subnet Mask and assign it to the “vSAN” portGroup

esxcfg-vmknic -a -i 192.168.100.1 -n 255.255.255.0 -p vSAN

We validate our VMKernel Interface by running the following command:

[root@se-emea-vsan01:~] esxcfg-vmknic -l
Interface? Port Group/DVPort/Opaque Network??????? IP Family IP Address????????????????????????????? Netmask???????? Broadcast?????? MAC Address?????? MTU???? TSO MSS?? Enabled Type??????????????? NetStack
vmk0?????? Management Network????????????????????? IPv4????? 172.16.101.1??????????????????????????? 255.255.252.0?? 172.16.103.255? 44:a8:42:29:fe:98 1500??? 65535???? true??? STATIC????????????? defaultTcpipStack
vmk1?????? vSAN??????????????????????????????????? IPv4????? 192.168.100.1?????????????????????????? 255.255.255.0?? 192.168.100.255 00:50:56:6a:5d:06 1500??? 65535???? true??? STATIC????????????? defaultTcpipStack

In order to add the VMKernel interface to vSAN we need to run the following command:

esxcli vsan network ip add -i vmk1

Repeat the above steps on the two remaining hosts that you wish to participate in the cluster

Step 2 – Creating the cluster
Once we have all the VMKernel interfaces configured on all hosts, we now need to create a vSAN Cluster on the first host, to do this we run the following command

esxcli vsan cluster new

Once completed we can get our vSAN Cluster UUID by running the following command:

[root@se-emea-vsan01:~] esxcli vsan cluster get
Cluster Information
?? Enabled: true
?? Current Local Time: 2016-11-21T15:17:57Z
?? Local Node UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
?? Local Node Type: NORMAL
?? Local Node State: MASTER
?? Local Node Health State: HEALTHY
?? Sub-Cluster Master UUID: 582a2bba-0fd8-b45a-7460-a0369f749a0c
?? Sub-Cluster Backup UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
?? Sub-Cluster UUID: 52bca225-0520-fd68-46c4-5e7edca5dfbd
?? Sub-Cluster Membership Entry Revision: 6
?? Sub-Cluster Member Count: 1
?? Sub-Cluster Member UUIDs: 582a29ea-cbfc-195e-f794-a0369f7894c4
?? Sub-Cluster Membership UUID: d2dd2c58-da70-bbb9-9e1a-a0369f749a0c

Step 3 – Adding the other nodes to the cluster
From the remaining hosts run the following command adding them to the newly created cluster

esxcli vsan cluster join -u 52bca225-0520-fd68-46c4-5e7edca5dfbd

You can verify that the nodes have successfully joined the cluster by running the same command we ran earlier noting that the Sub-Cluster Member Count has increased to 3 and it also shows the other sub cluster UUID Members:

[root@se-emea-vsan01:~] esxcli vsan cluster get
Cluster Information
?? Enabled: true
?? Current Local Time: 2016-11-21T15:17:57Z
?? Local Node UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
?? Local Node Type: NORMAL
?? Local Node State: MASTER
?? Local Node Health State: HEALTHY
?? Sub-Cluster Master UUID: 582a2bba-0fd8-b45a-7460-a0369f749a0c
?? Sub-Cluster Backup UUID: 582a29ea-cbfc-195e-f794-a0369f7894c4
?? Sub-Cluster UUID: 52bca225-0520-fd68-46c4-5e7edca5dfbd
?? Sub-Cluster Membership Entry Revision: 6
?? Sub-Cluster Member Count: 3
?? Sub-Cluster Member UUIDs: 582a29ea-cbfc-195e-f794-a0369f7894c4, 582a2bf8-4e36-abbf-5318-a0369f7894d4, 582a2c3b-d104-b96d-d089-a0369f78946c
?? Sub-Cluster Membership UUID: d2dd2c58-da70-bbb9-9e1a-a0369f749a0c

Step 4 – Claim Disks
Our cluster is now created and we need to claim the disks in each node to be used by vSAN, in order to do this we first of all need to identify which disks are to be used by vSAN as a Cache Disk and as Capacity Disks, and obviously the number of disk groups, to show the disk information for the disks in the host run the following command:

esxcli storage core device list

This will produce an output similar to the following where we can identify the NAA ID for each device:

naa.500003965c8a48a4
?? Display Name: TOSHIBA Serial Attached SCSI Disk (naa.500003965c8a48a4)
?? Has Settable Display Name: true
?? Size: 381554
?? Device Type: Direct-Access
?? Multipath Plugin: NMP
?? Devfs Path: /vmfs/devices/disks/naa.500003965c8a48a4
?? Vendor: TOSHIBA
?? Model: PX02SMF040
?? Revision: A3AF
?? SCSI Level: 6
?? Is Pseudo: false
?? Status: on
?? Is RDM Capable: true
?? Is Local: true
?? Is Removable: false
?? Is SSD: true
?? Is VVOL PE: false
?? Is Offline: false
?? Is Perennially Reserved: false
?? Queue Full Sample Size: 0
?? Queue Full Threshold: 0
?? Thin Provisioning Status: yes
?? Attached Filters:
?? VAAI Status: unknown
?? Other UIDs: vml.0200000000500003965c8a48a450583032534d
?? Is Shared Clusterwide: false
?? Is Local SAS Device: true
?? Is SAS: true
?? Is USB: false
?? Is Boot USB Device: false
?? Is Boot Device: false
?? Device Max Queue Depth: 64
?? No of outstanding IOs with competing worlds: 32
?? Drive Type: physical
?? RAID Level: NA
?? Number of Physical Drives: 1
?? Protection Enabled: false
?? PI Activated: false
?? PI Type: 0
?? PI Protection Mask: NO PROTECTION
?? Supported Guard Types: NO GUARD SUPPORT
?? DIX Enabled: false
?? DIX Guard Type: NO GUARD SUPPORT
?? Emulated DIX/DIF Enabled: false

In my setup I want to create two disk groups per host consisting of 4 capacity devices plus my cache so to create one disk group I run the following command:

esxcli vsan storage add -s <naa for cache disk> -d <naa for capacity disk 1> -d <naa for capacity disk 2> -d <naa for capacity disk 3> -d <naa for capacity disk 4>

Once you have performed the above on each of your hosts, your vSAN cluster is deployed with storage and you can now deploy your vCenter appliance onto the vSAN datastore where then you can manage your vSAN License, Storage Policies, switch on vSAN Services such as iSCSI, health service and performance services as well as start to deploy virtual machines

Configuring iSCSI on vSAN 6.5

One of the new features of vSAN 6.5 is iSCSI access, so before I talk about how to configure iSCSI on vSAN let me just cover what is meant by iSCSI on vSAN to avoid any confusion 🙂

As we know, vSAN is an object based storage system, everything that exists on vSAN is an object that has a storage policy applied to is via Storage Policy Based Management, in vSAN 6.2 the vSAN performance metrics are stored on vSAN as an object also, so vSAN is already gearing up for other types of objects and not just a file system for running virtual machines.? The iSCSI side of vSAN is no different, the iSCSI Target has a home folder object just like a virtual machine, and each iSCSI LUN created has it’s own object, just like a VMDK does for a virtual machine.? The main use cases for vSAN are as follows:

  • Physical Workloads
  • In virtual machine Software iSCSI
  • Supports MPIO
  • Microsoft Cluster Service (MSCS) and Windows Failover Cluster Instance (FCI) (as at the time of writing this, the MSCS/FCI will be supported in a future release)

VMware does not allow the presenting of iSCSI LUNs on vSAN to other vSphere clusters, when I sit back and think about that statement I can see a few reasons for this

  • There is no SATP in ESXi for vSAN iSCSI
  • You lose the whole functionality Storage Policy Based Management for your virtual machines
  • You end up defeating the object of vSAN and start placing all of your Virtual Machines back onto LUNs again

 

So how do we configure iSCSI on vSAN?? Well just like the whole simplicity around vSAN, iSCSI is no different and is extrmely simple to set up, for the purpose of the test I used a Windows 2012 Server Virtual Machine and the iSCSI Initiator Service in Windows

Step 1 – Create an iSCSI VMKernel Interface
This interface will be used for iSCSI communication to allow the iSCSI traffic to flow over to our Physical or even Virtual Machines

step-1-iscsi-vmkernel-interface

Step 2 – Enable the iSCSI Service on vSAN

step-2-enable-iscsi-target-service

Suring the enabling you are presented with various options:

  • VMKernel Interface
  • TCP Port Number
  • Authentication Method (CHAP and Mutual CHAP suupported)
  • The Storage Policy to be applied to the Home Object for the iSCSI Target

step-3-select-vmkernel-interface

 

Step 4 – Create an iSCSI LUN
For this I am creating an iSCSI LUN with a storage policy I have specified where my iSCSI LUNs are 100% Space reserved as per my Storage Policy, when creating the LUN it gives you an example on the right hand side of what this looks like from a vSAN Perspective, since I am using RAID1 – FTT=1, my 40GB LUN Object will consume 80GB of space and will be 80GB Reserved as per the policy

step-4-create-iscsi-target-and-lun

After this my display shows the LUN as being created and storage policy compliant

step-4b-lun-display

Step 5 – Add an Initiator or Initiator Group to the LUN Object
Before the Virtual Machine can see the LUN, I need to add the Initiator for the VM into the Allowed Initiators, I copied the initiator name from my virtual machine and pasted it in here

step-5-add-initiator

And the display shows this as now being added

step-5a-add-initiator

 

From a vSAN perspective this is the iSCSI configuration completed, so now I need to go to my Windows virtual machine and configure the iSCSI Initiator side of things, so the first thing I need to do is enable MPIO for iSCSI, this is listed in Windows 2012 under Administrative Tools, ensure the “Add support for iSCSI devices” is selected

step-6-enable-mpio-for-iscsi

Once I have MPIO configured for iSCSI Devices it is time to add my iSCSI Target, for this you need to go into Adminisrative Tools and select the iSCSI Initiator service, once in there go to the “Discovery Tab” and enter the IP Address of your iSCSI Target, in my setup the iSCSI Target was being hosted by host 2 and for this the iSCSI IP is 192.168.50.2

step7-add-portal

 

Configure any advanced settings for example if you are using CHAP/Mutual CHAP, then click on the “Targets” tab and you will see your Target IQN as “Inactive”, click on Connect

step7a-add-portal

In here select “Enable Multi Path”

step7b-add-portal

Once connected I can now see my 40GB LUN in Device Manager and also in Disk Management, in Device Manager it is listed as “VMware Virtual SAN Multi-Path Disk Device”…..yes I know it’s no longer called Virtual SAN

step-8-device-manager

step-9-disk-management

 

That’s your iSCSI Configured on vSAN and also connected the LUN to a Windows 2012 Server, like I said, very easy to do

Virtual SAN – Easy to Use Proactive Tests

Since VMware introduced Virtual SAN back in March 2014 there have been a number of enterprise class features and enhancements, one of which was the Health UI, incorporated into the Health UI is a section called Proactive Tests which can be used to perform functionality testing and performance testing, there are three areas for the Proactive Tests they are:

  • VM Creation Test
  • Multicast Performance Test
  • Storage Performance Test
Proactive Tests
Proactive Tests Screenshot

The tests themselves are a good way to test that your cluster configuration is correct and complete, so what does each one of the tests do?

VM Creation Proactive Test

The VM Creation test is a simple deployment of a small sized virtual machine on each host within the cluster you have created, the test is designed to validate that:

  • Most aspects of the Virtual SAN configuration are completed
  • The whole management stack and Virtual SAN network is correctly defined and configured

The results of the test is passed through to the UI to help identify potential mis-configuration issues if the test fails to complete, in order for the result to be passed through as a pass, all hosts must return a success.

Multicast Performance Proactive Test

We all know that part of the networking requirements for Virtual SAN is Multicast, it is required from a cluster management perspective and not from a data flow perspective, it is pretty difficult if you are not in charge of your network infrastructure to perform a Multicast test on the network to make sure all is correct.? This test will actually perform a multicast test between all hosts and measure the speed, there are three levels of status with the multicast test and they are:

  • Failed – One or more hosts produced a multicast performance test of 20MB/Sec or below
  • Warning – One or more hosts produced a multicast test result of More than 20MB/Sec but less than 50MB/Sec
  • Passed – All hosts produced a multicast performance test result exceeding 50MB/Sec

Like the VM Creation test, the results are passed through to the UI depending on the above success criteria.? This allows you to check that your network supports multicast as well as the performance of multicast on the physical network.

Storage Performance Proactive Test

I personally like playing around with this one in my all-flash lab with different RAID levels and different performance profiles, I also use this a lot with customers just after they have set up their environment for a proof of concept, it allows you to get a baseline of performance for the cluster and troubleshoot any mis-configuration prior to running the proof of concept that could affect the performance.

Within this test, you can select the duration of the test, the storage I/O profile as well as the policy that is to be used for the storage performance test which is a huge advantage to be able to test different storage policies and the performance that they will give based on the definitions you specify in each policy

VSAN Proactive Storage Text

So apart from being able to choose how long to run the test for and the storage policy used for the test, what about the workload itself?? Well in the workload there are a number of tests that you can perform, there are enough tests to be able to generate a workload to simulate most real world workloads out there today, under the tests I have performed, it is pretty clear that the Storage Performance test here is pretty I/O intensive and after running the tests a few times it was clear that the tests also use 4K block sizes.? So what are the options in the Workload types?

Basic Tests

  • Low Stress Test – This workload will deploy a single disk object per host in the cluster and perform just a generic low utilization stress test on all the disk objects at the same time, this is one of the only test that utilizes a single object per host for testing and is not very I/O intensive
  • Basic Sanity Test, focus on Flash cache layer – Just like the Low Stress Test this will also deploy a single object per host and perform a sanity test which will as it states, focus on the cahe layer of the Virtual SAN storage, this test is more suited to Hybrid where there is a read cache tier

Intensive Tests

Apart from the two tests above, all the rest of the tests are designed to be I/O intensive and will show what capabilities your Virtual SAN cluster will deliver, each of the workloads below will deploy mutiple objects per host within the cluster and perform the test on them all at the same time, so be warned:

  • Stress Test – Like the Low Stress Test, this test will perform the same test but with 20 objects per host as opposed to a single object per host in the Low Stress Test and will also use a I/O block size of 8K
  • Performance characterization – 100% Read, optimal RC usage – This test is designed to test the performance of the read cache in a Hybrid cluster
  • Performance characterization – 100% Write, optimal WB usage – This test is deigned to test the write buffer in both Hybrid and All-Flash clusters
  • Performance characterization – 100% Read, optimal RC usage after warmup – This test will perform the same as the Optimal RC usage test, however it will not perform the test until the cache has been warmed up first, this will allow Virtual SAN to see what blocks are regularly being accessed in order to cache them
  • Performance characterization – 70/30 read/write mix, realistic, optimal flash cache usage – For most workloads this will be the test most commonly used
  • Performance characterization – 70/30 read/write mix, realistic, High I/O Size, optimal flash cache usage – This test uses a 64K block size for I/O testing
  • Performance characterization – 100% read, Low RC hit rate / All-Flash Demo – This will show the real Read performance of your All-Flash environment very well, I have used this test many times in my All-Flash environment
  • Performance characterization – 100% streaming reads – A test that would be the equivalent of streaming content from Virtual SAN via multiple sessions
  • Performance characterization – 100% streaming writes – A test that would be the equivalent of multiple video surveillance cameras streaming data to Virtual SAN

After the test has completed, the results will be displayed in the window below the test options like the following, as you can work out from my results below, the test I performed was for the High I/O Size as the MB/s divided by IOPS gives a block size of around 64K

VSAN Storage Performance Results

If we look at out performance charts for the same period of time we can see that the IOPS was over 88,000 for the cluster at 64K Block Size and a throughput of 5.4GB/second (43.2Gbps)

VSAN 64K IOPS Graphs

If we do the same for the All-Flash Demo these are the results based on the profile “Performance characterization – 100% read, Low RC hit rate / All-Flash Demo”

VSAN Storage Performance Results-AF

Again if we look at out performance charts for the same period of time we can see that the IOPS was over half a million for the cluster at 4K Block Size

VSAN IOPS Graphs-AF

Customers always ask me how realistic these tests are, I have done some comparisons with HCIBench also, and so have some of my customers, and HCIBench produces the same results, so as you can see the Virtual SAN Proactive Tests can save you a lot of time and effort with regards to things like storage performance testing, and remember they are very I/O intensive, coupled with the new performance monitoring charts in Virtual SAN, you have more tools at your fingertips all within a single user interface.

Enjoy 🙂

 

 

 

 

 

 

It's all about VMware vSAN