Since VMware introduced Virtual SAN back in March 2014 there have been a number of enterprise class features and enhancements, one of which was the Health UI, incorporated into the Health UI is a section called Proactive Tests which can be used to perform functionality testing and performance testing, there are three areas for the Proactive Tests they are:

• VM Creation Test
• Multicast Performance Test
• Storage Performance Test

The tests themselves are a good way to test that your cluster configuration is correct and complete, so what does each one of the tests do?

VM Creation Proactive Test

The VM Creation test is a simple deployment of a small sized virtual machine on each host within the cluster you have created, the test is designed to validate that:

• Most aspects of the Virtual SAN configuration are completed
• The whole management stack and Virtual SAN network is correctly defined and configured

The results of the test is passed through to the UI to help identify potential mis-configuration issues if the test fails to complete, in order for the result to be passed through as a pass, all hosts must return a success.

Multicast Performance Proactive Test

We all know that part of the networking requirements for Virtual SAN is Multicast, it is required from a cluster management perspective and not from a data flow perspective, it is pretty difficult if you are not in charge of your network infrastructure to perform a Multicast test on the network to make sure all is correct.? This test will actually perform a multicast test between all hosts and measure the speed, there are three levels of status with the multicast test and they are:

• Failed – One or more hosts produced a multicast performance test of 20MB/Sec or below
• Warning – One or more hosts produced a multicast test result of More than 20MB/Sec but less than 50MB/Sec
• Passed – All hosts produced a multicast performance test result exceeding 50MB/Sec

Like the VM Creation test, the results are passed through to the UI depending on the above success criteria.? This allows you to check that your network supports multicast as well as the performance of multicast on the physical network.

Storage Performance Proactive Test

I personally like playing around with this one in my all-flash lab with different RAID levels and different performance profiles, I also use this a lot with customers just after they have set up their environment for a proof of concept, it allows you to get a baseline of performance for the cluster and troubleshoot any mis-configuration prior to running the proof of concept that could affect the performance.

Within this test, you can select the duration of the test, the storage I/O profile as well as the policy that is to be used for the storage performance test which is a huge advantage to be able to test different storage policies and the performance that they will give based on the definitions you specify in each policy

So apart from being able to choose how long to run the test for and the storage policy used for the test, what about the workload itself?? Well in the workload there are a number of tests that you can perform, there are enough tests to be able to generate a workload to simulate most real world workloads out there today, under the tests I have performed, it is pretty clear that the Storage Performance test here is pretty I/O intensive and after running the tests a few times it was clear that the tests also use 4K block sizes.? So what are the options in the Workload types?

Basic Tests

• Low Stress Test – This workload will deploy a single disk object per host in the cluster and perform just a generic low utilization stress test on all the disk objects at the same time, this is one of the only test that utilizes a single object per host for testing and is not very I/O intensive

• Basic Sanity Test, focus on Flash cache layer – Just like the Low Stress Test this will also deploy a single object per host and perform a sanity test which will as it states, focus on the cahe layer of the Virtual SAN storage, this test is more suited to Hybrid where there is a read cache tier

Intensive Tests

Apart from the two tests above, all the rest of the tests are designed to be I/O intensive and will show what capabilities your Virtual SAN cluster will deliver, each of the workloads below will deploy mutiple objects per host within the cluster and perform the test on them all at the same time, so be warned:

• Stress Test – Like the Low Stress Test, this test will perform the same test but with 20 objects per host as opposed to a single object per host in the Low Stress Test and will also use a I/O block size of 8K

• Performance characterization – 100% Read, optimal RC usage – This test is designed to test the performance of the read cache in a Hybrid cluster

• Performance characterization – 100% Write, optimal WB usage – This test is deigned to test the write buffer in both Hybrid and All-Flash clusters

• Performance characterization – 100% Read, optimal RC usage after warmup – This test will perform the same as the Optimal RC usage test, however it will not perform the test until the cache has been warmed up first, this will allow Virtual SAN to see what blocks are regularly being accessed in order to cache them

• Performance characterization – 70/30 read/write mix, realistic, optimal flash cache usage – For most workloads this will be the test most commonly used

• Performance characterization – 70/30 read/write mix, realistic, High I/O Size, optimal flash cache usage – This test uses a 64K block size for I/O testing

• Performance characterization – 100% read, Low RC hit rate / All-Flash Demo – This will show the real Read performance of your All-Flash environment very well, I have used this test many times in my All-Flash environment

• Performance characterization – 100% streaming reads – A test that would be the equivalent of streaming content from Virtual SAN via multiple sessions

• Performance characterization – 100% streaming writes – A test that would be the equivalent of multiple video surveillance cameras streaming data to Virtual SAN

After the test has completed, the results will be displayed in the window below the test options like the following, as you can work out from my results below, the test I performed was for the High I/O Size as the MB/s divided by IOPS gives a block size of around 64K

If we look at out performance charts for the same period of time we can see that the IOPS was over 88,000 for the cluster at 64K Block Size and a throughput of 5.4GB/second (43.2Gbps)

If we do the same for the All-Flash Demo these are the results based on the profile “Performance characterization – 100% read, Low RC hit rate / All-Flash Demo”

Again if we look at out performance charts for the same period of time we can see that the IOPS was over half a million for the cluster at 4K Block Size

Customers always ask me how realistic these tests are, I have done some comparisons with HCIBench also, and so have some of my customers, and HCIBench produces the same results, so as you can see the Virtual SAN Proactive Tests can save you a lot of time and effort with regards to things like storage performance testing, and remember they are very I/O intensive, coupled with the new performance monitoring charts in Virtual SAN, you have more tools at your fingertips all within a single user interface.

Enjoy 🙂

 

 

 

 

 

 

Out of all the conversations I have surrounding Virtual SAN with Customers, Partners and VMware folks, there is one topic that always surfaces and that topic is with regards to what drivers and firmware to use, is it the minimum version on the compatibility guide or what?? As we know Virtual SAN has a specific HCL when it comes down to three components

• Disk Controller (VSAN RAID0 and Passthrough)
• Solid State Drives
• Magnetic Disks

So where does the confusion come from?

Historically, the compatibility guide for ESXi has always been supported as what was certified by the vendor, and anything newer than that is perfectly acceptable, but in the case of Virtual SAN this all changed, especially for the disk controller, and this is due to the whole interaction between Virtual SAN and the controller that the slightest change in firmware could result in a behavioural change on the controller which could have negative consequences on the Virtual SAN environment.? With Virtual SAN the hardware vendors will certify specific combinations of drivers and firmware, for example here

You will see that the controller listed here for ESXi 5.5U3 (Virtual SAN 5.5) has two different drivers and associated firmware versions, this is the explicit combination that is required, so for example it would not be supported to use the driver version highlighted in red megaraid_perc9 version 6.901.55.00.1vmw with the firmware version highlighted in blue 25.3.0.0016 and vice versa, this is the exact firmware combination that must be used.? So if you update the firmware of the controller, you need to also ensure you apply the correct driver version also.

So what about newer versions you might ask?? Sometimes vendors will release newer Drivers and Firmware to provide fixes to known issues or enhancements to the controller, until that newer driver/firmware has been certified for Virtual SAN you should not upgrade to them, the chances are that it is in the process of being certified but until you see it on the Virtual SAN Compatibility Guide do not upgrade.? Stay Safe…..Stay Supported!

With regards to the Solid State Drives and Magentic disks, on the Virtual SAN Compatibility Guide you will see a firmware version published (Not in all cases for Magnetic disks), like the traditional ESXi Compatibility Guide, this is the minimum firmware supported, so as vendors produce newer firmware for them, you can update them as you wish, just remember to be on or above the mimumum published on the Virtual SAN Compatibility Guide

So just remember that:

• Disk Controller – The exact driver / firmware combination on the Virtual SAN Compatibility Guide
• Solid State Disks and Magnetic Disks – The firmware published on the Virtual SAN Compatibility Guide is the minimum version supported

Virtual SAN Compatibility Guide – Component Based

Sometimes during a deployment you might not be able to or want to install ESXi to the embedded SD Card controller on the Dell PowerEdge servers, for example if your memory configuration exceeds 512GB of RAM then installing ESXi to an SD Card would not be supported, or you may want to have a locally defined Scratch Partition on a local VMFS volume rather than using a remote Syslog or NFS Share, so what are the options?

Well you could boot ESXi from a single disk, the downside to that is if the disk fails then you are in a host down situation, the other option is to use two disks in a RAID1 mirror and use that for the ESXi installation.? With servers such as the R730XD you can utilise all 24 drive slots in the front of the server for Virtual SAN, and then two rear 2.5 Inch slots for your RAID 1 Mirror drives.? In my lab environment I have a similar config, I have a bunch of SSD drives used by Virtual SAN and then two 300GB 10K SAS Drives in a RAID 1 Mirror for my ESXi install and local VMFS Volume.? So how do we configure the H730 controller to support both Passthrough Disks and RAID disks?

Please note: For production environments this type of configuration is not supported

The first thing we need to do is ensure that the controller itself is configured for RAID Mode, for those that do not know, the H730 controller can be configured in either RAID Mode or HBA Mode.? RAID Mode allows you to create RAID Virtual Disks to be handled by the Logical Disk Controller as part of a RAID Volume, it also allows you to configure physical disks as NON-RAID disks, HBA Mode allows you to configure disks as NON-RAID capable only, in order to set the controller as RAID Mode we need to enter the System Setup Screen, to do this reboot the host and press F2 to enter the System Setup Main Menu, once there we need to enter the Device Settings option

 

This section of the System Setup will allow you to change the settings for any devices connected in the system, for the purpose of this article we will be focusing on the PERC H730 controller, in my servers I have two controllers, so I am selecting the one that is second in the list labelled RAID Controller in Slot 3: Dell PERC <PERC H730P Adapter> Configuration Utility

 

Once in the Device settings we need to chose the option for Controller Management and at the way to the bottom and choose the option for Advanced Controller Management, in there you will see an option for Switch to RAID Mode, when selected it will inform you that a reboot is required, do not reboot the host just yet.

 

Now we need to ensure that NON-RAID Disk Mode is enabled, for this, click on Back and enter the Advanced Controller Properties at the bottom of the screen and select the option to enable for Non RAID Disk Mode and hit Apply Changed

 

At this point you will need to make the disks you wish to use as a RAID1 as RAID Capable, you can do this under Configuration Management and choose the option to Convert to RAID Capable, from within here select the two disks you wish to use as your RAID 1 and click on OK, check the Confirm box and click on yes to confirm

 

Click on Back to go to the main menu and then re-select Configuration Management and chose the option to Create Virtual Disk and proceed to create your RAID 1 disk, it will only let you select disks that have been switched to RAID capable, after completion, check that all other disks are seen as NON-RAID Disks under Physical Disk Management, since we switched the controller from HBA Mode to RAID Mode all the other disks should still be tagged as NON-RAID

Now your PERC H730 controller is configured for both RAID and Passthrough and your RAID1 Virtual Disk can be used to install your ESXi, during the ESXi installation, make sure you install to the correct disk, and all your other disks are passthrough mode and can be consumed by Virtual SAN