VMware vCenter 6.7 U1: Windows to VCSA Upgrade and Convergence

Today we will be talking about the VMware vCenter 6.7-U1 (Update 1) upgrade process. I recently had an opportunity to work with a enterprise customer to upgrade their VMware environment. In this post we will be going through the upgrade process and my thoughts. VMware 6.7 U1 is a major upgrade that includes the fully featured HTML5 client. For full details on what’s new please see: https://blogs.vmware.com/vsphere/2018/10/whats-new-in-vcenter-server-6-7-update-1.html

I will start by saying bravo to the VMware team for this release. For the first time I actually felt comfortable abandoning the good ol’ “fat client” (the legacy C# client). Many of VMware’s customers, in my experience, were intentionally lagging behind on older versions of vCenter to keep a cold death-grip on the fat client because they refused to be force-fed the flash client that we all know and despise. The HTML5 client is a worthy successor. It’s fast, it looks good, its organized better, and it even has a dark mode. It’s obvious they took feedback from the community, hired the right developers who understood their target audience, and put out a great product. The upgrade and migration process is also done very well.

After a few weeks of the VCSA and HTML5 client baked into the client environment it’s obvious that some things are still missing, like exporting events, from the HTML5 client but I would expect these to be eventually added. There also appears to be some lag to the recent tasks list in larger linked environments. I’ve also seen a few UI bugs with adding permissions and modifying sDRS configuration.

One issue I’ve seen on multiple VCSA’s so far is that the database “archive” (disk 13) will constantly fill up causing the VCSA to show up as degraded within the dashboard. You will be greeted with the error message “File system /storage/archive is low on storage space. Increase the size of disk /storage/archive.” There is very little documentation on this but apparently this is expected behavior despite the warnings and rational I don’t quite understand yet. This didn’t stop me from increasing the disk size (KB2126276) slightly.

The 6.7-U1 Upgrade Experience

Like most customers running VMware your vCenters are probably running Windows, you have an external platform services controller (PSC), and probably even separate database servers. Well as of 6.7-U1 you will be making some changes to the topology of your vCenter deployment. VMware now recommends the embedded platform service controller topology for all new and existing deployments. But what does this mean? In one word: simplified. The environment will have less complexity and less dependency on other systems. The integrated database means no need for a local or external database which introduced performance and latency hits as well as complicates recovery during an outage. The embedded platform services controller for each VCSA means no need to point to an external PSC which for some customers just ended up being a single point of failure.

Below are some notes I made for administrators considering an upgrade. Your starting point should be the VMware 6.7 Upgrade Guide.

My Recommendations:

Migrate Windows vCenter to VCSA: VMware has openly said support for Windows will be going away and that VCSA is now the primary platform.

Converge to Embedded PSC: This is the new VMware recommended deployment topology. There are situations where this is not appropriate and external may still be best. Be aware that you cannot converge a VCSA to embedded with a historical data import job queued, running, or in a hung state. The historical data import must complete or be cancelled in order to converge.

Upgrade datastores to VMFS6: Your environment may benefit from VMFS6, particularly if you use SSD or NVMe based storage. VMFS6 is 4K aligned in order to support newer Advanced Format (AF) large-capacity drives among many other improvements. WARNING: You cannot upgrade a VMFS5 datastore to VMFS6. You must migrate, remove and re-add the datastore to format as VMFS6. Your ESXi hosts must be at least 6.5.x or higher.

Pre-run the Migration Assistant prior to migration/upgrade day: This will expose any issues before you upgrade. Note that the assistant will terminate when it encounters an issue and does not proceed with the rest of it’s checks. This means you need to keep running the Migration Assistant until you get to the state where it is ready. The log files will be your primary resource to troubleshooting these issues.

Pre-build the converge scripts: You should plan-out and pre-build the converge.json and decommission_psc.json templates.

Historical Data Import: If you’re migrating a large vCenter then use the background historical data import. This will allow the vCenter services to start without having to wait a potentially lengthy period.

Your skills may be tested: A fresh install will be very straight forward but the migration from Windows to VCSA can become very complicated very quickly. Be prepared for the unexpected. I personally ran into many undocumented issues during an upgrade from a Windows Server 2008 R2 vCenter. This was the customers development vCenter and we expected this one to be smooth sailing, but it turned into a nightmare that required quick thinking and our skills were certainly put to the test during it’s upgrade.

In conclusion I highly recommend an upgrade to 6.7 Update 1, but don’t rush into it. Ensure you take the time to plan your upgrade.

vSphere 6.7 U1 now released

On October 17, 2018 VMware announced that vSphere 6.7 Update 1 is now available. The new HTML5 client is now ‘Fully Featured’ which means that you can use the HTML5 client for all administration and configuration of vSphere; including Auto Deploy, Host Profiles, VMware vSphere Update Manager (VUM), vCenter High Availability (VCHA), network topology diagrams, overview performance charts, and more.

I am personally excited to see the HTML5 client become the primary client as I much prefer using it over the flash client. One of the more interesting features included in this release is the vCenter External to Embedded Convergence tool. Since embedded PSC is the recommended deployment model for vCenter Server this tool allows you to migrate to an embedded PSC without having to nuke-and-pave your entire vCenter installation.

The Content Library also got some much needed love from the VMware development team as it now supports two more new file formats; allowing templates and OVA files. This makes the Content Library much more functional. The lack of VM templates was a major caveat of the Content Library to the point of making it practically useless for some VMware customers. So this change is a welcome one to say the least.

New Features

  • vCenter High Availability (VCHA)
    • We redesigned VCHA workflows to combine the Basic and Advanced configuration workflows. This streamlines the user experience and eliminates the need for manual intervention of some deployments.
  • Search Experience
    • We revamped the search experience. In this version of the vSphere Client, you can now search for objects with a string and filter the search results based on Tags/Custom attributes. You can also filter the object lists in the search even further. For instance, you can filter on the power state of the VMs etc., You can save your searches and revisit them later.
  • Performance Charts
    • You can pop the performance charts into a separate tab and zoom in on a specific time in the chart. We also added overview performance charts for datacenters and clusters.
  • Dark Theme
    • Dark theme has been one of the most requested features for the vSphere Client so we’re introducing a Dark mode setting. Support for the Dark theme is available for all core vSphere Client functionality and implementation for vSphere Client plugins is in progress.
  • Alarm Definitions
    • We greatly simplified the way you define new alarms, particularly in how you create rules for trigger conditions.

VMUG Advantage is 10% off

Today VMware announced that VMUG Advantage is 10% off until December 31st, 2017.

This is the best way to get VMware licenses for your home lab environment. This is included with VMUG Advantage membership through EVALExperience which gives you a 365-day evaluation license for personal use in a non-production environment. Continue reading…

vSAN all hosts down scenario

 

The worst case scenario in a VMware vSAN cluster is all hosts down. A situation where no sysadmin wants to find themselves in. Panic & frustration quickly follow suit. Despite all the safety features built into vSAN it is designed to tolerate failures within it’s failure domains, not an entire vSAN cluster outage.

Scenario

Unsaid client was in the process of setting up a VDS on an existing VSAN cluster. Mistakenly selected the vSAN vmkernel adapters on all hosts for migration to the VDS while the cluster was in operation. Upon deploying this change it instantly took down the entire 4-node, 14TB vSAN cluster. All VMs down, vSAN data store showing as 0KB. To add to the mix, the customers vCenter VCSA was also down because it was also hosted on the vSAN which made it even more difficult to view the overall health of the environment.

  • vSphere 6.5 environment
  • vSAN total failure, non-stretched, single host failure domains
  • All vSAN VMs down including vCenter VCSA
  • 4-node cluster vSAN
  • Hybrid disk groups (1 flash, 2 HDD per host)
  • NumberOfFailuresToTolerate=1

Disaster Recovery

This is a cluster network total failure. This results in a complete network partition of vSAN where each host will reside in its own partition. To each isolated host, it will look like all the other hosts have failed. Since no quorum can be achieved for any object, no rebuilding takes place. Once the network issue is resolved vSAN will try to establish a new cluster and components will start to resync. Components are synchronized against the latest, most up to date copy of a component.

Continue reading…

Migration from Cisco 1000v to VMware Virtual Distributed Switch (Part 2)

home_network3

This is part 2 of a series. Click here to see Part 1. I apologise for taking so long to get Part 2 posted. Sometimes I just don’t have the time or effort I would like to have with the blog.

000193_2015-10-29 10_06

This portion of the guide focuses on the second half of the VSS to VDS migrations. We needed to move the VMs to a VSS so that you can migrate both VMs and hosts to the new vCenter cleanly. Then we will be moving the VMs back to a VDS from their VSS configuration.

Keep in mind this migration is being done LIVE with production virtual machines running on the hosts. Obviously, this must be executed carefully or you will have a lot of explaining to do. Do not make these changes without understanding the full impact to your environment. Continue reading…

vExpert 2016

vmware_hyk

I am very honoured to be selected as a vExpert 2016 by VMware. Getting recognition is awesome but knowing that you are sharing content that is for the benefit of others is even better.

The annual VMware vExpert title is given to individuals who have significantly contributed to the community of VMware users over the past year. The title is awarded to individuals (not employers) for their commitment to sharing their knowledge and passion for VMware technology above and beyond their job requirements.

vExpert’s benefits and activities receive:

  • vExpert certificate
  • Permission to use the vExpert logo on cards, website, etc for one year
  • Access to a private directory for networking, etc.
  • Exclusive gifts from various VMware partners
  • Access to private betas (subject to admission by beta teams)
  • 365-day eval licenses for most products
  • Private pre-launch briefings
  • Private briefings from tier 1 alliance partners
  • Blogger early access program for vSphere and some other products
  • Featured in a public vExpert online directory
  • Access to vetted VMware & Virtualization content for your social channels.

I give thanks to the other vExperts and the VMware social media & community team for their hard work and dedication.

The full list of the 2016 vExperts can be found here.

VMW-LOGO-vEXPERT-2016-k