Every upgrade within vSphere major versions requires some pre-upgrade tasks to first be completed. Those are always third party software and hardware updates. It is often a bad idea to upgrade too fast when you use a lot of third party software. Your backups, management tools, and hardware may fail when in use. Therefore, it is always best to plan your upgrade tasks for when these subsystems are ready to upgrade.
In this case, I had VMware software in addition to hardware and third party upgrades to make. I wanted to ensure my VMware management tools were at the latest version before I upgraded the underlying vCenter and ESXi nodes. Why? To make those upgrades faster and be more integrated.
VMware Software Stack
For example, my version of vCenter (v6.7) allows for vRealize Suite 2019 to be in use. Upgrading the 8.x versions of vRealize Operations, vRealize Log Insight, and vRealize Automation allows me to use newer functions with my older stack but prepares for the next version of vSphere as well.
So in essence, what you can pre-upgrade before you upgrade vSphere within the VMware software portfolio, limits the amount of changes at any one time. Small changes are best, not large changes. Never the complete stack at once. This limits the overall impact of any one upgrade.
Third Party Software
My main third party software components are Veeam Availability Suite and HPE OneView Suite. Each of these suites contain more than one software component that needs upgrading. The longest hold out of my third party software was HPE OneView Suite. It took them a while to officially support vSphere 7.x.
Upgrading my software suite for Veeam was pretty uneventful. It was necessary. Yet, at the same time simple. Veeam does a very good job with updates.
HPE Oneview on the other hand took reinstalling HPE Oneview for vCenter package. The other two components upgraded seamlessly. HPE Oneview for vCenter reconfiguration was pretty simplistic so it was no great effort to reinstall. It was however unexpected.
During all these third party software the approach to upgrade is to always take a snapshot first, do the upgrade, test, then commit (delete all) snapshots on success. Or on failure to revert to previous snapshot and start again or wait until the bug is fixed fully.
Bugs with upgrades often will cause delays in upgrading the entire stack. It is best therefore to do third party upgrades out of sync with vSphere upgrades as long as the previous vSphere version is fully supported. If not, plan to do third party upgrades AFTER your vSphere upgrade.
Hardware Updates
Hardware updates often sound dangerous, but in this case they were not. Usually, I take this opportunity to just upgrade everything to the latest firmware. However, on this particular upgrade I was moving from microSD cards for my boot volume to M.2 SSD cards for my boot volumes. Yes, firmware was upgrade, but because of the move to M.2 I was required to reinstall ESXi.
I could have done this as part of my vSphere 7.x deployment but then I would be mixing and matching 6.x with 7.x and that would cause some difficulty. Such major changes are often best when everything is on the same version of vSphere? Why? Because, even though host profiles is limited, host profiles is still a extremely useful tool in this situation.
Migrating Boot Volumes
To migrate boot volumes it is required to get Host Profiles setup first. Well maybe not required, but it sure saves time to recreate your entire configuration. You can do it by hand. I choose not to do so. Host Profiles for v6.7 is pretty good if you disable a few things to make it better to use with your pre-upgrade tasks. This items to disable are:
- Advanced Configuration Settings -> Configuration Files -> vCenter agent (vpxa) configurations (this is from pre 6.7 configurations inherited from upgrades)
- General System Settings -> Device Alias Configuation -> Device Alias Configuration -> any alias setting for vmhba devices
- General System Settings -> Management Agent Configuration -> CIM Indication Subscriptions (this is from old CIM subscriptions)
- Security and Services -> Security Settings -> Security -> Role -> CIMUser (this is from an old CIM tool)
- Security and Services -> Security Settings -> Security -> User Configuration -> CIMUser (this is from an old CIM tool)
- Storage Configuration -> Pluggable Storage Architecture (PSA) configuration
- Storage Configuration -> Native Multi-Pathing (NMP)
- Storage Configuration -> Software FCoE Configuration
As you can see, many of these items to disable are related to Storage or older CIM tools inherited from previous upgrades. The new installs will no longer contain the historical CIM settings. Yet, they will continue to contain Storage, even with identical systems, this is often an issue with Host Profiles. As a matter of course to ensure storage issues do not appear throughout your profile checks.
So what do we gain from Host Profiles? Not the storage configuration, as that will be inherited by the install onto the M.2 cards. We will NOT be changing any other local or remote datastores with this. In particular, host profiles will configure our virtual networking for VSS or VDS, as well as iSCSI, NFS, and VSAN.
Since I have a lot of networks, this is a big win. Recreating everything by hand is a royal pain. However, even with host profiles, 3 problems occurred. The solution was to use existing resiliency capabilities to automatically fix. Knowing these will aid in your pre-upgrade planning and recovery.
vmnic overlap
When you use host profiles to recover the network configuration of the host, vmnic0 is used for administration. That is fine, but vmnic0 is assigned to a VSS not a VDS. To fix this, we need to ensure vmk0 is on the proper VDS. Fix this and every other VDS recovers automatically.
VXLAN vmkernel recreate
The vmkernel port for VXLAN is recreated automatically, but when Host Profiles does this, VXLAN cannot make use of the vmkernel port. The solution is to remove the Host Profiles created vmkernel port. At that time NSX will automatically recreate the VXLAN vmkernel port for you.
VSAN vmkernel port
I also had issues with the VSAN vmkernel port. It was not joining the network properly so VSAN looked disjoint. The solution was to remove the vmkernel port created by Host Profiles and allow it to be recreated by VSAN itself.
Maintaining Host Profiles
Now comes the age old question, do we maintain host profiles, or do we ignore any issues (or disconnect the host from Host Profiles). I have chosen to manage them. The only change so far I have seen is a need to update the esxupdate firewall rule to be enabled. I did an upgrade after reinstalling my hosts from ISO. So that is to be expected.
There are probably a bunch of unknown changes coming related to the next upgrade of tools. Will I keep Host Profiles or not? That depends on what I need to do. Having a list of what to disable for the next use is the most important thing. This will help during my next set of pre-upgrade tasks.
Conclusion
While upgrading firmware and third party software is often straight forward. My pre-upgrade tasks were complexed by the need to move off microSD to M.2. MicroSD cards will go bad suddenly. M.2 is a bit more stable a resource. Yet, I also have to upgrade to 7.x soon, which means everything else needs to be in order first.
This pre-upgrade was mostly about hardware. All the other upgrades happened over many weeks. Small perterbations are best for your environment, not large ones. If you do have large ones, like new boot volumes, ensure you have the tooling to recover quickly.
