By the end of 2017 nearly all my NAS servers were close to reaching full capacity. I had already pre-decided on getting a Synology DS1817+ but it was just a matter of when. I wanted something that was more than five bays and would be upgradable to 10G networking in the future. The DS1817+ seemed to match all of my needs and my budget. Continue reading…
It’s been long overdue for some changes to my home lab. The latest full outage on Sept 4, 2017 due to a power brown-out had me realizing that some improvements can be made. There has not been any major changes to the lab since 2015. In 2016 I upgraded the storage in NAS1, memory upgrade for VMH02, added Ubiquiti UAP-AC-LITE access points, and a security camera.
Now I’m going back to the drawing board and doing a fresh rebuild. The goal this time around is to be simple and redundant.
- Hardware firewall: I have custom built a 1U Supermicro server that will be used as the new firewall. It has a Intel Xeon X3470 CPU, 8GB RAM, quad gigabit LAN ports and a 200W low power supply. I’ve also replaced the stock passive CPU heat-sink with the Thermaltake Engine 27 low profile heat-sink. It’s a well balanced combination of performance, power and noise. In the old lab design the virtualized firewall introduced too many dependencies and greatly increased the complexity of the network. During a power outage scenario it also requires me to have a VM host and storage online which does not last long on UPS batteries. Having a low power hardware firewall allows me more flexibility and faster recovery from a total lab black-out.
- Additional UPS backup power: There will now be a third UPS battery for the home lab. I will dedicate one UPS for the core networking equipment and try to keep the load on it under 25% to maximize the battery life. The rest of the gear will be balanced over the other two UPS batteries.
- Standard Virtual Switches: I will be removing the Virtual Distributed Switch and LACP on the ESXi hosts. This is a tough call but I have weighed the options. The VDS in my environment is overkill. I have two hosts, with only one of them on at a time. In my scenario the VDS’s only purpose is configuration sync. I don’t use traffic shaping, private VLANs, LLDP, etc! The only loss I will take by moving down to a VSS is having to manually maintain the port groups exactly the same on each host and no LACP. That doesn’t concern me because that hardly ever changes.
This past weekend we had a power brownout for about 4 hours. This caused my servers to fail-over to battery power. The batteries don’t last long with servers running. I guess something went sour with the automatic shutdown of my NAS3 which is used only for my VMware virtual machines and it did an improper shutdown. The RAID has crashed.
I don’t have anyone to blame other than myself and I knew eventually this day would come. NAS3 was in RAID-0. That means striping with no redundancy. A failed array on RAID-0 typically means total data loss. I take daily backups of this entire NAS nightly so I am aware and prepared for the risk of using striping. That does not mean that it’s a fun time recovering from it.
Adding additional redundancy for blackouts
Currently, one of the hardest things to recover from in my current home-lab environment is a total power blackout. Everything right now is planned & designed around losing certain components like 1 disk, 1 switch/network cable, etc. However when everything is off and I need to bring things back online it’s a painstaking and very manual process. Over time my environment has also become more and more complex. This latest outage has me scratching my head at how to recover faster & simpler from a power blackout.
I’ve been meaning to make a post about all the recent changes to my home lab but I’ve been quite busy. I’ve also done some more work on the backend of the website to help speed things up. I’m also, slowly, working on a new design for vSkilled as well.
The biggest update I have right now is that I’ve finally ordered a portable air conditioning unit for my home lab. It’s starting to get warmer again since summer is around the corner and I don’t want the house to be ridiculously warm. I ordered the Honeywell 12,000 BTU MN12CES. Once I have the unit installed I’ll try and put up another post with a write up and pics!
I have come to the conclusion that in 2017 I will need to down scale my home lab in order to reduce power & cooling usage. It’s grown year over year and unless I start making changes it’s not going to start going down.
My plan is to beef up VMH02 with more RAM so that it can handle the full load of the VMs. Then I will have VMH01 powered-off in stand-by mode. This way only one of the ESXi hosts are running at a time but can still quickly spin up when needed using VMware power control with IPMI if needed. This should reduce my power usage in the lab significantly, especially because both of my ESXi servers are dual CPU socket – they love to eat up power. Having only one of the servers running should make a huge difference. I have never used VMware power management before so I am both curious and excited to make use of it. Continue reading…
I recently switched over from Sophos UTM to Untangle NG for my personal use firewall at home. During the process I basically had to rebuild all of my firewall rules and general network policy configurations. This allowed to me “start fresh” as my previous configuration had gotten quite bloated and complicated over time.
It’s clear that Microsoft has no intentions of telling us what exactly is sent in this telemetry data, how long it’s stored, and why when it’s disabled it continues to send data. Not to mention which obvious third parties have access to the data. For this reason, part of the new network policies I wanted to include was blocking telemetry data from getting sent back to the Microsoft mother-ship. Continue reading…