vSphere Upgrade Saga: Installing HP StoreVirtual 11.0

I have had HP StoreVirtual (LeftHand) licenses for some time now, but the previous versions were nearly impossible to install. At least, I could not discover the magic incantation to make them work, so I decided to put that upgrade on hold. Now that 11.0 has come out, as have a host of installers for various hypervisors, I will be giving it another shot. Assuming it installs, it will allow an additional 4TBs to be available to my infrastructure. I am running pretty close to the limits on my current datastores.

First attempt…

Here are the steps I followed to install a single, non-redundant, split-network HP StoreVirtual. In split-network installations, one network is used for management and one network is used by iSCSI traffic.

Step 0: Download all the HP StoreVirtual bits; there are quite a few

The most important ones are the Failover Manager OVF and the StoreVirtual OVF bits. You will need both. I also downloaded the entire document set, as I wanted to have a split network. This, I found out, does not really work well, but more on that later.

Step 1: Create a user and role for HP StoreVirtual within vCenter

This is one of those security items. Each tool that talks to vCenter requires its own user and role. Unfortunately, HP and VMware often require specific roles, such as Administrator, in order to do certain things. I may give them the Administrator role during install to get past that stage but then change their roles to their own.

Step 2: Install the HP StoreVirtual

The graphical UI installer for HP StoreVirtual died on me twice with no error message, so I switched to the CLI installer, which gave me a bit more information. Apparently, the installer is looking for the LHN.vim.Distributedportgroup. This does not exist on my systems, so I can only assume these are really updaters, not installers. Installation also failed using the Installer EXE and the VSA ISO.

So, I deployed the VSA from OVF following the instructions in the HP StoreVirtual Storage VSA Installation and Configuration Guide. When it came time to configure the virtual disk, I pointed it to the DS2200sb that is in my blade enclosure and increased the memory per the instructions in the documentation. Since I currently have vSphere 5.1, I am limited to 2TB virtual disk volumes, which is a bummer until I upgrade to vSphere 5.5.

Lastly, there are two network adapters. I used the first for administration and configured the second for iSCSI. The latter is on my iSCSI portgroup of a dVS.

Step 3: Install the StoreVirtual Management Software

I have a separate system for all HP management products, so I installed the software onto this and not onto vCenter or any other node. I used the HP StoreVirtual Management ISO as the basis for the install.

Step 4: Connect the VM to the StoreVirtual Centralized Management Console (CMC)

This was automatic! There was no need to do anything but launch the CMC after install.

Step 5: Create a management cluster

This is the tricky part, as we have two NICs defined, one for management and one for iSCSI. The best approach is to set up a split-network configuration. However, you can only do that after creating the management cluster.

This is accomplished in two steps: (1) Change the virtual IP address to be the subnet of the storage network. (2) Change the LeftHand OS interface under the storage menu element under the storage system to be the second network adapter (eth1).

This allows for management via one network and for storage to be served up on another.

Step 6: Add servers to the StoreVirtual CMC

HP StoreVirtual will not serve up any storage until the servers are allowed to do so. This is done by adding the server, specifying the vCenter server, and entering the associated IQN by hand within the CMC of the HP StoreVirtual management group. For large environments, entering all this data would be tiresome. I would assume that HP StoreVirtual would work within the HP Insight Control for vCenter.

Step 7: Activate Active Directory integration for management

This step caused me a bit of trouble until I found several posts by Craig Kilburn over at vmfocus.com. These solved my issues. My main problem was trying to use the wrong login notation. HP StoreVirtual requires UPN notation for all logins and is case sensitive. So, if your domain is DOMAIN.local, using a login of login@domain.local will not work. It needs to be login@DOMAIN.local; Something to consider when using Active Directory integration.

Step 8: Register the HP StoreVirtual within HP Insight Control for vCenter Storage Manager

Visit the HP Insight Control for vCenter -> Storage Administrator Portal for vCenter and add the HP StoreVirtual to the list using the user defined in Step 6.

Note that there is a problem with this, however. The iSCSI subsystem uses one subnet for iSCSI traffic, and the CMC uses another subnet for management. There is a virtual IP that points to the iSCSI subsystem, but there is none for the StoreVirtual Management, and once you enter one IP, the iSCSI IP shows up. Not even vCenter should be able to see the iSCSI storage directly. This is a problem with a split-network configuration.

If you do allow vCenter to talk directly to the iSCSI network, then Insight Control for vCenter’s storage does show up properly.

Step 9: Patch HP StoreVirtual

Within the CMC, you can go to Configuration Summary and determine whether there are any patches for your version of HP StoreVirtual by clicking on the Upgrades tab. I upgraded to all the critical patches.

HP StoreVirtual is now available. While there are a few changes I would like to make (including adding a second disk storage node for redundancy), I at least have something for now.

After a month or So…

Not too long after I finished the above work, I received my second HP StoreVirtual License and enough disk to make another HP StoreVirtual VSA node. This changes my architecture somewhat, as I really need all the features of vCenter, HP Insight Control, and Veeam’s backup software to work correctly. I continued down the path of a split network using the following steps:

Step 1: Install the second VSA

This is the second VSA node, to add redundancy, so I performed steps 2, 4, and 9 from the previous list. However, that was not enough for the second VSA to join the cluster. To make a cluster work, you need a third node, at the very least, to act as an arbiter in case of a split decision on whether or not a node is alive: a Failover Manager (FOM).

Step 2: Deploy Failover Manager from OVF

Extremely straightforward, but it has to live on the iSCSI network. Split network does not work very well for the FOM except for contact via the CMC.

Step 3: Add FOM to management group

The FOM is a specialized manager that becomes part of the management group. The key to remember is that its communication path is over the iSCSI network. Ensure that the LeftHand OS interface is the same across all nodes.

Step 4: Add the second VSA

Once more, everything worked as expected. I did have a few issues with mismatches between my LeftHand OS interfaces, but once this was solved, everything functioned smoothly. I now had 4TB of storage in a Network Raid-10 configuration with two-way mirroring.

Things would have been fine if all I wanted was storage, but I also wanted to take advantage of HP Insight Control for vCenter and HP StoreVirtual’s VASA provider, and I wanted Veeam to have the ability to back up from the HP StoreVirtual Snapshots. These tools require the ability to talk to the iSCSI network, so I put in place a gateway firewall device to limit access to this network to just those tools. However, since I am using a split network, authentication and routing took place over the management network, not the iSCSI network, which causes all these tools to fail unless the systems in question are directly on the iSCSI network. You should never place your management systems directly on the iSCSI network.

Final implementation…

After some conversations with HP support and Rick Vanover of Veeam, I decided that the routing issues were too great to overcome using a split network. If the HP StoreVirtual cannot route the iSCSI network properly, then there is no hope of getting all these tools to work properly. I tried the following to no avail:

• Added routes to the firewall.
• Added routes on the iSCSI interface to specific nodes, which should be able to talk to the iSCSI interface. This worked to satisfy iSCSI, but it failed on logging in, as all traffic from the node to the StoreVirtual then went through that route, including authentication traffic.

So, the split-network design does not work very well. Since I had no data on my HP StoreVirtual, and it is pretty straightforward to start over, I went ahead and reinstalled everything, with a few changes:

• Increased the size of the HP StoreVirtual to 3TB (now that my nodes are at vSphere 5.5)
• Installed without a split network.
• Routed all iSCSI traffic and HP StoreVirtual traffic through a vCNS Edge firewall.

Step 1: Install from OVF

The first step of my redeploy was to deploy two VSAs and one FOM from their respective OVFs. I kept them powered off, as I wanted to ensure that the networking and associated hard disks were correct.

Step 2: Deploy vCNS Edge

Using the vShield Manager, because there isn’t a way to deploy vCNS Edge from the vSphere Web Client, and the vSphere .NET Client integration seems to be broken, I deployed a vCNS Edge device that sits between my administrative network and the network associated with iSCSI devices. First, I created an iSCSI portgroup for virtual machines that is on the vSwitch associated with the iSCSI vmKernel devices. This way, they can all talk to each other.

Step 3: Associate networks on StoreVirtual VMs with the iSCSI network

The next step was to ensure that the three VMs associated with the HP StoreVirtual (FOM, two VSAs) were properly associated with the iSCSI portgroup. This happens as a part of installation, but it is best to double-check the networking.

Step 4: Move VSA VMs and associate storage

I had to migrate the VSA VMs to the appropriate nodes, to which HP D2200SB devices were attached. These devices act as local storage. Once the VSA VMs were migrated, I created a hard disk 3TB in size on each of the devices. Previously, I had created a VMFS on each of the D2200SBs.

Step 5: Power on each VM and associate IP addresses

I created a brand-new DNS domain for my HP StoreVirtual systems, just so they show up in DNS properly, and I associated the appropriate network and IP with them. To do this, open up the VMware Remote Console and follow the instructions on the screen; then, start the management interface and log in. Once you log in, it is pretty easy to associate an IP, subnet mask, and gateway with each of the three nodes of the HP StoreVirtual. The gateway in this case was the iSCSI side of the vCNS Edge device.

Step 6: Ensure management nodes have appropriate route

VMware vCenter, HP Insight Control, Veeam, and Active Directory systems need a route to the iSCSI network so that those nodes can talk through the vCNS Edge device to the managers of the HP StoreVirtual.

Step 7: Associate each VM with the CMC

Using the “Getting Started” link of the CMC, I was able to “Find the Systems” by entering their IP addresses manually (they are on a different subnet with a firewall between the management node and the actual HP StoreVirtual components). Next, I used the “Management Groups, Clusters, and Volumes Wizard” to create the HP StoreVirtual within the CMC.

Step 8: Fix up administration of the HP StoreVirtual

This step is a catchall to fix any additional issues or items, such as:

• Enable Active Directory integration
• Enable Email integration
• Enable SNMP integration
• Associate servers with the HP StoreVirtual (ESXi nodes and Veeam node)
• Fix any time server issues (I had one due to a routing issue, but changing the time server fixed the problem)
• Perform any upgrades (there was one upgrade pending for me)

Step 9: Add to HP Insight Control for vCenter

This is the first step in integrating HP StoreVirtual VASA with vCenter. Using the HP Insight Control storage administrator, I was able to create an entry pointing to the HP StoreVirtual. Then, I was able to get the HP StoreVirtual VASA provider address and, under Storage Profiles, add it to vCenter. This works out pretty well, actually. However, you need to ensure that the node running HP Insight Control for vCenter can talk to the iSCSI network through the vCNS Edge.

Step 10: Integrate Veeam Backup and Restore

This step took place in two parts on the Microsoft Windows machine running Veeam Backup and Replication. The first part was to connect the Windows iSCSI Initiator to the HP StoreVirtual. Once that was successful, you can move onto add an HP StoreVirtual into Veam Backup and Restore. Please remember that the user in question requires full_administrator or view_only_full_administrator access within the HP StoreVirtual CMC.

The key here is to ensure the Veeam Backup and Replication VM or node can talk through the vCNS Edge to reach the HP Store Virtual.

Now I have a fully integrated HP StoreVirtual that is not only redundant, but does two-way mirroring between separate nodes. This means I can finally start using my new storage!