VMware vCenter 6.7 U1: Windows to VCSA Upgrade and Convergence

Today we will be talking about the VMware vCenter 6.7-U1 (Update 1) upgrade process. I recently had an opportunity to work with a enterprise customer to upgrade their VMware environment. In this post we will be going through the upgrade process and my thoughts. VMware 6.7 U1 is a major upgrade that includes the fully featured HTML5 client. For full details on what’s new please see: https://blogs.vmware.com/vsphere/2018/10/whats-new-in-vcenter-server-6-7-update-1.html

I will start by saying bravo to the VMware team for this release. For the first time I actually felt comfortable abandoning the good ol’ “fat client” (the legacy C# client). Many of VMware’s customers, in my experience, were intentionally lagging behind on older versions of vCenter to keep a cold death-grip on the fat client because they refused to be force-fed the flash client that we all know and despise. The HTML5 client is a worthy successor. It’s fast, it looks good, its organized better, and it even has a dark mode. It’s obvious they took feedback from the community, hired the right developers who understood their target audience, and put out a great product. The upgrade and migration process is also done very well.

After a few weeks of the VCSA and HTML5 client baked into the client environment it’s obvious that some things are still missing, like exporting events, from the HTML5 client but I would expect these to be eventually added. There also appears to be some lag to the recent tasks list in larger linked environments. I’ve also seen a few UI bugs with adding permissions and modifying sDRS configuration.

One issue I’ve seen on multiple VCSA’s so far is that the database “archive” (disk 13) will constantly fill up causing the VCSA to show up as degraded within the dashboard. You will be greeted with the error message “File system /storage/archive is low on storage space. Increase the size of disk /storage/archive.” There is very little documentation on this but apparently this is expected behavior despite the warnings and rational I don’t quite understand yet. This didn’t stop me from increasing the disk size (KB2126276) slightly. [2019-04-12: This issue is now fixed by VMware.]

The 6.7-U1 Upgrade Experience

Like most customers running VMware your vCenters are probably running Windows, you have an external platform services controller (PSC), and probably even separate database servers. Well as of 6.7-U1 you will be making some changes to the topology of your vCenter deployment. VMware now recommends the embedded platform service controller topology for all new and existing deployments. But what does this mean? In one word: simplified. The environment will have less complexity and less dependency on other systems. The integrated database means no need for a local or external database which introduced performance and latency hits as well as complicates recovery during an outage. The embedded platform services controller for each VCSA means no need to point to an external PSC which for some customers just ended up being a single point of failure.

Below are some notes I made for administrators considering an upgrade. Your starting point should be the VMware 6.7 Upgrade Guide.

My Recommendations:

Migrate Windows vCenter to VCSA: VMware has openly said support for Windows will be going away and that VCSA is now the primary platform.

Converge to Embedded PSC: This is the new VMware recommended deployment topology. There are situations where this is not appropriate and external may still be best. Be aware that you cannot converge a VCSA to embedded with a historical data import job queued, running, or in a hung state. The historical data import must complete or be cancelled in order to converge.

Upgrade datastores to VMFS6: Your environment may benefit from VMFS6, particularly if you use SSD or NVMe based storage. VMFS6 is 4K aligned in order to support newer Advanced Format (AF) large-capacity drives among many other improvements. WARNING: You cannot upgrade a VMFS5 datastore to VMFS6. You must migrate, remove and re-add the datastore to format as VMFS6. Your ESXi hosts must be at least 6.5.x or higher.

Pre-run the Migration Assistant prior to migration/upgrade day: This will expose any issues before you upgrade. Note that the assistant will terminate when it encounters an issue and does not proceed with the rest of it’s checks. This means you need to keep running the Migration Assistant until you get to the state where it is ready. The log files will be your primary resource to troubleshooting these issues.

Pre-build the converge scripts: You should plan-out and pre-build the converge.json and decommission_psc.json templates.

Historical Data Import: If you’re migrating a large vCenter then use the background historical data import. This will allow the vCenter services to start without having to wait a potentially lengthy period.

Your skills may be tested: A fresh install will be very straight forward but the migration from Windows to VCSA can become very complicated very quickly. Be prepared for the unexpected. I personally ran into many undocumented issues during an upgrade from a Windows Server 2008 R2 vCenter. This was the customers development vCenter and we expected this one to be smooth sailing, but it turned into a nightmare that required quick thinking and our skills were certainly put to the test during it’s upgrade.

In conclusion I highly recommend an upgrade to 6.7 Update 1, but don’t rush into it. Ensure you take the time to plan your upgrade.

Karl has been involved in the virtualization, server, web development and web hosting industry for over 15 years. In his current role at a managed service provider, he is focused on cloud-based solutions for enterprise clients. His diverse background of sales, management, and architectural/technical expertise bring a unique perspective to the virtualization practice.