vSAN all hosts down senario

 

The worst case scenario in a VMware vSAN cluster is all hosts down. A situation where no sysadmin wants to find themselves in. Panic & frustration quickly follow suit. Despite all the safety features built into vSAN it is designed to tolerate failures within it’s failure domains, not an entire vSAN cluster outage.

Scenario

Unsaid client was in the process of setting up a VDS on an existing VSAN cluster. Mistakenly selected the vSAN vmkernel adapters on all hosts for migration to the VDS while the cluster was in operation. Upon deploying this change it instantly took down the entire 4-node, 14TB vSAN cluster. All VMs down, vSAN data store showing as 0KB. To add to the mix, the customers vCenter VCSA was also down because it was also hosted on the vSAN which made it even more difficult to view the overall health of the environment.

  • vSphere 6.5 environment
  • vSAN total failure, non-stretched, single host failure domains
  • All vSAN VMs down including vCenter VCSA
  • 4-node cluster vSAN
  • Hybrid disk groups (1 flash, 2 HDD per host)
  • NumberOfFailuresToTolerate=1

Disaster Recovery

This is a cluster network total failure. This results in a complete network partition of vSAN where each host will reside in its own partition. To each isolated host, it will look like all the other hosts have failed. Since no quorum can be achieved for any object, no rebuilding takes place. Once the network issue is resolved vSAN will try to establish a new cluster and components will start to resync. Components are synchronized against the latest, most up to date copy of a component.

Continue reading…

Migration from Cisco 1000v to VMware Virtual Distributed Switch (Part 2)

home_network3

This is part 2 of a series. Click here to see Part 1. I apologise for taking so long to get Part 2 posted. Sometimes I just don’t have the time or effort I would like to have with the blog.

000193_2015-10-29 10_06

This portion of the guide focuses on the second half of the VSS to VDS migrations. We needed to move the VMs to a VSS so that you can migrate both VMs and hosts to the new vCenter cleanly. Then we will be moving the VMs back to a VDS from their VSS configuration.

Keep in mind this migration is being done LIVE with production virtual machines running on the hosts. Obviously, this must be executed carefully or you will have a lot of explaining to do. Do not make these changes without understanding the full impact to your environment. Continue reading…

Virtual Firewall and Networking – Planning Guide

This is a planning guide on how to create a robust, redundant, virtual network for your home-lab environment including a virtual firewall. This requires a lot of existing hardware and expertise. This is not recommended the faint of heart and will challenge you. Using a physical firewall is the easy choice.

Cisco_Nexus_3000_Series_1

I have structured this guide around how I have my own network configured for the vSkilled home lab. I have been running in this configuration for literally years without incident. You should first weigh the pros and cons for your own environment and then decide if this design is the right choice for YOU. Just because it works for me, does not mean it will work for you. There are many mixed opinions between running your firewall physically or virtually. Neither is right or wrong. That really depends entirely on your skill level and the equipment you have available. You should decide on a network topology which you are most comfortable troubleshooting and fixing when it breaks.

Continue reading…

Windows Server 2016 – Technical Preview 5

Install

Microsoft has released Windows Server 2016 Technical Preview 5 (build #14300). You can see what’s new here. This could probably be one of the last few TP (tech preview) builds that we will see. Especially so if Microsoft is still firm on their plans to officially launch Windows Server 2016 this summer (Q3 2016).

As long as you are running VMware ESXi 5.5 or higher (6.0 or later is recommended) then Windows Server 2016 is an supported operating system on VMware. You can even select it as an option for the guest OS on virtual machine version 11 or higher. Keep in mind that VMware VM version 11 restricts you to using the web client ONLY. When moving from a previous version of Windows Server to Windows Server 2016 Technical Preview 5, you will need to uninstall the previous version for a clean installation of Technical Preview 5. You can download TP5 as an ISO, however Nano server is only available in VHD format. See Getting Started with Nano Server for full details.

Personally I was never a fan of Windows 8.x or Server 2012. So far I think that Windows Server 2016 is already step in a better direction. Even in technical preview it offers many improvements of it’s predecessor. Windows Server 2016 Technical Preview 5 provides a wide range of new and enhanced features and capabilities spanning server virtualization, storage, software-defined networking, server management and automation, web and application platform, access and information protection, virtual desktop infrastructure, and more. The GUI version or what is now referred to by Microsoft as the “Desktop Experience” is my current de-facto standard. If you use or have seen Windows 10 then right out of the box you will notice that Server 2016 is a stripped down, optimized, server version of Windows 10.

Choose Standard or Datacenter edition, depending on the features you need:

  • Windows Server 2016 Standard
    • Up to 2 VM’s or Hyper-V containers
    • Unlimited Windows containers
    • New Nano Server deployment option for “just enough OS”
  • Windows Server 2016 Datacenter
    • Unlimited VM’s and Hyper-V containers
    • Unlimited Windows containers
    • New Nano Server deployment option for “just enough OS”
    • Shielded VM’s and Host Guardian Service
    • Storage features, including Storage Spaces Direct and Storage Replica
    • New networking stack

Windows Server 2016 Technical Preview 5 Gallery:

Overall Technical Preview 5’s new features seem to be focused on Hyper-V, Networking, Storage, Nano Server and Security. In Server 2016 you will also find Windows Defender and “Windows Server Antimalware” is installed and enabled by default.

The introduction of Host Guardian Service (HGS)’s new feature Shielded Virtual Machines which focuses on the security of virtual machines running in the Hyper-V environment. The goal of shielded VMs and Guarded Fabric is to help provide service providers and cloud operators the ability to offer their tenant administrators a hosted environment where protection of tenant virtual machine data is strengthened against threats from compromised storage, network, and host administrators, as well as malware.

This is just a quick post showcasing the new tech preview build. I will have a more in-depth view of all of these features and more when a release candidate build is finally available.

What do you think of Windows Server 2016 so far? Comment below!

vExpert 2016

vmware_hyk

I am very honoured to be selected as a vExpert 2016 by VMware. Getting recognition is awesome but knowing that you are sharing content that is for the benefit of others is even better.

The annual VMware vExpert title is given to individuals who have significantly contributed to the community of VMware users over the past year. The title is awarded to individuals (not employers) for their commitment to sharing their knowledge and passion for VMware technology above and beyond their job requirements.

vExpert’s benefits and activities receive:

  • vExpert certificate
  • Permission to use the vExpert logo on cards, website, etc for one year
  • Access to a private directory for networking, etc.
  • Exclusive gifts from various VMware partners
  • Access to private betas (subject to admission by beta teams)
  • 365-day eval licenses for most products
  • Private pre-launch briefings
  • Private briefings from tier 1 alliance partners
  • Blogger early access program for vSphere and some other products
  • Featured in a public vExpert online directory
  • Access to vetted VMware & Virtualization content for your social channels.

I give thanks to the other vExperts and the VMware social media & community team for their hard work and dedication.

The full list of the 2016 vExperts can be found here.

VMW-LOGO-vEXPERT-2016-k

Migration from Cisco 1000v to VMware Virtual Distributed Switch (Part 1)

Cisco_Nexus_3000_Series_1
While working with a enterprise customer I was tasked with migrating an entire production environment from the Cisco Nexus 1000v to a VMware Virtual Distributed Switch (VDS). Then moving the VDS and the ESXi 5.1 hosts over to a fresh built vSphere 6.0 server. The customer is in the middle of an upgrade from vCenter 5.1 to 6.0. Most of the host upgrades will be done once the hosts are moved over to to the new vCenter.

Goals:

  • Non-disruptive migration of networking for Virtual Machines (this is a live production environment)
  • Migrate away from the Cisco 1000v, to VDS
  • Migrate the VDS config from old 5.1 vCenter to new 6.0 vCenter
  • Touch-up naming of virtual machine networks/VLANs
  • Move Virtual Machines from the Virtual Distributed Switch (VDS)/Nexus 1000v to a Virtual Standard Switch (VSS)
  • Disconnect and remove the ESXi hosts from the old vCenter 5.1
  • Connect ESXi hosts to the new vCenter 6.0
  • Migrate VM networking from VSS to VDS

VMware vSphere 5.1 and later allow you to export, import, or restore Distributed Switch configurations from the vSphere Web Client. Since moving the 1000v would be too convoluted, if not actually impossible, I will move everything over to a VDS on the existing 5.1 vCenter first. Then once everything is up and running on the VDS we can then migrate the VDS configuration over to the new 6.0 vCenter server.

000193_2015-10-29 10_06

Unfortunately we will also need to create a Virtual Standard Switch (VSS) switch configured with all the networks, all with matching configuration on each ESXi host in order to actually do the ESXi host migrations over to the new 6.0 vCenter. This will be automated with scripts, of course. We must migrate all virtual machine, VMkernel, and service console networking from VDS to VSS so that network connectivity is not lost when we remove the hosts from the VDS in order to disconnect them from the 5.1 vCenter and add them to the 6.0 vCenter.

MIGRATION FROM 1000v to VDS

Summary:
The following steps will migrate the host and VM networking from the Cisco Nexus 1000v to a VMware Virtual Distributed Switch. This migration plan assumes that there are at least two dedicated uplinks for VM traffic. The purpose of this is to remove dependencies on the legacy 1000v and create a known working configuration of the VDS that will later migrated to the new vCenter 6.0 server. The customer has decided against using the 1000v and wants it removed from their environments. We need to perform this migration before we can move the hosts to the v6.0 vCenter so that we have a working VDS configuration that we can later export to the new vCenter and as a result have an immediately working VDS configuration.

I performed the migration in two parts; that I named “legs” which is basically a reference to the actual uplinks (A + B) themselves. This is to ensure I can quickly and easily roll-back the change if necessary. For the duration of the migration we will only be on one “leg” at a time (either the 1000v on uplink A – or – the VDS on uplink B), this of course introduces a single point of failure but the risk is acceptable since the change window is quite small and the chances of a switch or uplink failure during our change is low. Regardless, I will ensure that both the 1000v and VDS are fully working at all times until all VM networking is migrated from the 1000v to the VDS – testing along the way to ensure there is no impact to VM networking. Once all VMs are moved from the 1000v then we can remove the uplink to the 1000V which at that point should be completely unused and add that uplink to the VDS so that we can then achieve our A+B uplink redundancy again.

Pre Tasks:

  • *** Disable HA, DRS, and EVC on the Cluster ***
  • *** Storage DRS needs to be set to manual or disabled ***

LEG 1
The following will put the hosts into a split 1000v + VDS configuration. One leg on the 1000v, one leg on the VDS. This is necessary to allow proper configuration verification and full migration of VM networking. During this time however, VMs will only have 1 uplink on either side which introduces a single point of failure. However this will only be for the duration of the migration and then they will move back to 2+ uplink paths.

1 – Place target host into maintenance mode
2 – Remove ONE host uplink to the 1000v
3 – Attach host to new VDS using the now available vmnic that used to be on the 1000v
4 – Remove host from maintenance mode
5 – Use Testing VM to verify networking is working (at your discretion)
6 – Repeat process for all hosts individually

LEG 2
The following will migrate the 2nd leg into the uplink group of the first to allow proper link redundancy on the VDS. Currently VM networking should be working on both the 1000v and VDS. The following steps will fully disconnect the 1000v and migrate VM networking to the VDS. This will allow the removal of the 1000v while the VMs continue running without interruption.

1 – Using the ‘Migrate Virtual Machine Networking’ tool migrate ALL VM networks as required from the 1000v to the VDS networks
2 – Repeat the Migrate Virtual Machine Network for each VM Port Group until all VMs are migrated
3 – ** At this point all VMs should be running from a VDS **
4 – Place target host into maintenance mode
5 – Remove host from the 1000v
6 – Add the vmnic that was on the 1000v to the VDS uplinks
7 – Packet Control (etc) vmnics should show as DOWN on the host
8 – Remove host from maintenance mode
9 – Repeat process for all remaining hosts

Post Tasks:
– Re-enable HA, DRS, EVC, and Storage DRS as appropriate
– Export working VDS configuration to the new v6.0 vCenter server

We’re done!

This migration was completed successfully for the customer in all development, staging, and production VMware environments. During the migration we even took the time to clean-up and standardize the network names of the VM port groups consistently across the environments. I hope this guide can be helpful if you find yourself in a similar situation.

Click here for Part 2, where we will migration the VDS to VSS networking so that we can move the hosts to the new vCenter server, the move back to the VDS on the new vCenter 6.0!

If you have any comments, questions, or suggestions please let me know in the comments section below!

 

References / Sources: