Add large amounts of flash via ATTO XtreamCORE and JBOF

Remote flash for ESXi Hosts and read cache for VMs using JBOD with SSD (JBOF) and ATTO Technology XstreamCORE 7500 series storage controllers

Adding flash to any system can improve performance.  It’s usually getting the flash to the system that can be the challenge.  In a standard ESXi system if you have internal slots available you can add SSDs internally, if you have an external SAS connection, you can buy a JBOF shelf and have lots of SSD flash and drives available for that server.  But what if your bays are full and it does not make sense to buy 1 shelf per server plus add in a SAS HBA?  For a few hosts this might be the most cost effective answer.  For more than a few hosts, the options seem to be add in a large all flash array (I assume they virtualize all the drives and present them as remote flash either iSCSI or FC) or if there was a JBOF with FC connections that was cost effective…but then you have some limits as well since its one shelf.  ATTO Technology can resolve this problem.  TheXstreamCORE 7500 and 7550 storage controllers are high performance protocol bridging appliances (they can do a lot more but for this example it’s what they do).  Four 12Gb SAS controllers allow for many shelves of flash to be connected and are presented as Fibre Channel storage via 16Gb Fibre Channel ports. The screen shots below show an example of assigning a single SSD across a FC Fabric to a host as remote flash and what can be done with it.

Enabling 1 or more drives as flash for a host

Here we can add flash for the host:

And finally an example of adding read cache to a virtual drive in a VM

Virtualizing a MacPro and creating MacOS virtual machines

My personal compiled list of how to accomplish this.  Please note this compilation is intended for legal use of hardware and software and should be used within the Apple EULA.

Primary credit goes to William Lam @lamw and the resources he created and linked on Virtually Ghetto. Most of this is simply a summary of many of his blogs on the topic.

Step 1: Strong Recommendations to consider before you start:

This is pretty important and when I first started this project I skipped right to making an ESXi host without thinking “what do I need from this host BEFORE I overwrite the SSD”.  It was a huge mistake and added quite a bit of time to my overall discovery of what worked for me.  I would strongly recommend that you back up the MacOS on your MacPro before installing ESX onto the SSD.  I would also STRONGLY recommend you create your MacOS Boot media ISO that you will use for your VMs (setting that as next step) if this is the MacOS version you will install, prior to installing ESXi.

Step 2: Media prep for bootable MacOS media

You will need to log into an MacOS system which still has its install dmg on the system.  Where you see a name such as Sierra.app in the script below, you can change if you are using a different version of MscOS (ElCapitan.app as an example).  Run all of these commands in a terminal window.  Once complete, you MAY have some issues with the ISO not moving cleanly to a USB stick due to size and formatting issues of the USB (It will falsely report the iso being too large for the USB stick).   To resolve, format the USB stick as exFat with Rufus and this should allow you to move the ISO from the Mac to the machine you are using as a host iso file for the MacOS on a VM install.

hdiutil attach /Applications/Install\ macOS\ Sierra.app/Contents/SharedSupport/InstallESD.dmg -noverify -nobrowse -mountpoint /Volumes/install_app

hdiutil create -o /tmp/Sierra.cdr -size 7316m -layout SPUD -fs HFS+J

hdiutil attach /tmp/Sierra.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/install_build

asr restore -source /Volumes/install_app/BaseSystem.dmg -target /Volumes/install_build -noprompt -noverify -erase

rm /Volumes/OS\ X\ Base\ System/System/Installation/Packages

cp -rp /Volumes/install_app/Packages /Volumes/OS\ X\ Base\ System/System/Installation/

cp -rp /Volumes/install_app/BaseSystem.chunklist /Volumes/OS\ X\ Base\ System/BaseSystem.chunklist

cp -rp /Volumes/install_app/BaseSystem.dmg /Volumes/OS\ X\ Base\ System/BaseSystem.dmg

hdiutil detach /Volumes/install_app

hdiutil detach /Volumes/OS\ X\ Base\ System/

hdiutil convert /tmp/Sierra.cdr.dmg -format UDTO -o /tmp/Sierra.iso

mv /tmp/Sierra.iso.cdr ~/Desktop/Sierra.iso

 

Step 3: Media Preparation:

I personally download the ISO I wish to use as the install media from the VMware download site.  I use ESXi-Customizer to embed the drivers I want or need into it.  I would recommend once you download ESXi-Customizer (which is a dos program) make a script to call it up.  MUCH easier that using a DOS window!  Depending on the version of ESXi you want to use and the MacPro you may need to embed a SATA driver or worst case scenario is the need to embed an ethernet driver.  If you are going to create your ESXi media with drivers such as the ATTO Thunderlink Fibre Channel adapter, so that you could install to a remote LUN, you would want to do that in this step.  Once I have the ISO I will use, I use RUFUS to create a bootable USB stick to boot the host from.  NOTE if you are installing ESXi 6.5 or a version that does NOT prefer the vmklinux drivers you may have to do some extra steps.  See my caveats below.

 

Step 4: ESXi Installation:

Hold the Alt key if you are using a non Mac keyboard as you power on the MacPro with the USB stick inserted into a USB slot.  This lets you choose what device to boot the host with.  Follow the prompts and install as you would any other host.  HINT that the host nic 0 is the nic on the right and nic 1 is the nic on the left. I would recommend using vCenter and adding the host to your datacenter.

Step 5: Creating the VMs

VM creation is normal except select “Other” for the OS in the VM Creation and select the exact version of MacOS you will be creating or the highest version of MacOS listed if you are installing a higher version.

Step 6: Installing MacOS

I have uploaded the iso to a datastore and mounted it to the VM or I have used a local ISO mounting of a CD-ROM to the VM to install the MacOS instance.  I typically do this from the vCenter VM console viewer for that VM than attach media as a part of the VM creation.  It’s easier for me to disconnect and remember.  I usually download the vmware tools from the repository and manually use the dmg to install after the vm is up and running.

Caveats

  1. No Directpath IO  Sorry it is a limitation in the ability for ESXi to push it directly to the MacOS.  I have tried everything I can think of to make it work without success.
  2. Sometimes Apple changes drivers. When the ESXi installer cannot see the sata drive or the NIC on your initial install, it’s likely that the base drivers with ESXi cannot see the hardware.   Troubleshooting this is a bit of a pain.  My PERSONAL step is to try a different (newer or older) version of ESXi and when the problem is resolved simply upgrade to the newer version.
  3. ESXi 6.5 and the vmklinux issue. If you try to install on 6.5 where the vmklinux is not enabled by default (meaning the installer wants to use all native drivers) you will need to tweak your install media.  How do you know this is happening?  You’re your install fails at different points usually between the initial load and the actual install, then its probably this issue.  Think about it from the installer perspective.  I see most of the hardware but I cannot interact with some of it. What I usually do is the following (again TY William):  *** NOTE that I did NOT have to do this with 6.5U1 release.
    1. Use an unzip (I use 7Zip) program to open the .iso installer
    2. There will be 2 files to edit. They are in the root of the iso boot.cfg and /efi/boot/boot.cfg
    3. In these files will be a line (close to the top) that says kernelopt=runweasel You will need to append this line so that it says kernelopt=runweasel preferVmklinux=True
    4. Once you have saved the changes re-make the iso with an iso creator and you should be able to boot using this iso file and the previously mentioned steps.

 

 

Links to the above mentioned files and stuff you may need.  NOTE many of these tools are unsupported but simply make life easier for the non-powerCLI folks (like me)

 

ESXi-Customizer (A few caveats are on the page itself such as no Win 10 support)

https://www.v-front.de/p/esxi-customizer.html#download

 

RUFUS  I recommend the portable version

https://rufus.akeo.ie

Hello world!

So yes I have set up a blog and am working through things now about setting it up properly and organizing things to make it easier for you to see only what you want etc.  There will be a technology side and a personal side to all of this.  For not plan on technology being the focus.  Sometimes hard to mix work and life together…..

Eric Garrison