Home Lab Rebuild

It’s been long overdue for some changes to my home lab. The latest full outage on Sept 4, 2017 due to a power brown-out had me realizing that some improvements can be made. There has not been any major changes to the lab since 2015. In 2016 I upgraded the storage in NAS1, memory upgrade for VMH02, added Ubiquiti UAP-AC-LITE access points, and a security camera.

Now I’m going back to the drawing board and doing a fresh rebuild. The goal this time around is to be simple and redundant.

  1. Hardware firewall: I have custom built a 1U Supermicro server that will be used as the new firewall. It has a Intel Xeon X3470 CPU, 8GB RAM, quad gigabit LAN ports and a 200W low power supply. I’ve also replaced the stock passive CPU heat-sink with the Thermaltake Engine 27 low profile heat-sink. It’s a well balanced combination of performance, power and noise. In the old lab design the virtualized firewall introduced too many dependencies and greatly increased the complexity of the network. During a power outage scenario it also requires me to have a VM host and storage online which does not last long on UPS batteries. Having a low power hardware firewall allows me more flexibility and faster recovery from a total lab black-out.
  2. Additional UPS backup power: There will now be a third UPS battery for the home lab. I will dedicate one UPS for the core networking equipment and try to keep the load on it under 25% to maximize the battery life. The rest of the gear will be balanced over the other two UPS batteries.
  3. Standard Virtual Switches: I will be removing the Virtual Distributed Switch and LACP on the ESXi hosts.  This is a tough call but I have weighed the options. The VDS in my environment is overkill. I have two hosts, with only one of them on at a time. In my scenario the VDS’s only purpose is configuration sync. I don’t use traffic shaping, private VLANs, LLDP, etc! The only loss I will take by moving down to a VSS is having to manually maintain the port groups exactly the same on each host and no LACP. That doesn’t concern me because that hardly ever changes.

Continue reading…

Firewalls for Home Use

A question I see often is what firewall is the best for a home/residential environment? Before I get into that, we must realize that the majority of non tech-savvy people do not even have a firewall, or they have one but it’s not enabled/configured correctly, or they’re just not sure. In an age where we see more weaponized vulnerabilities and threats year after year – this is a huge problem. The problem though, is as big as an issue for consumers as it is for businesses such as ISPs and network device manufactures.

Home router firmware hasn’t change much over time. In early 2016, The Wall Street Journal looked at the security capabilities of the top 20 home routers. Only six of those had up-to-date firmware at that time, and just two of them had good password processes. The recent ASUS settlement with the Federal Trade Commision over the critical security flaws in their home routers is further proof that home router manufacturers don’t take security seriously. Today’s home router selections don’t offer you the flexibility to set up your network the way you see fit. They also don’t provide you visibility into the devices that are connecting to your network says Untangle.

There is a wide array of security practices that would probably make you shake your head.  Just the other day I was at my parents place and found that the ISP provided modem/gateway’s firewall was set to “NAT only”. The firewall was disabled and it even stated that this was the default option and that enabling the firewall was “optional”. I would highly suspect that this is the default configuration for all of the ISP’s customers. This means the firewall functionality and security legwork is responsibility of the end-device. Scary! Continue reading…

Home Labs: Remote Access and Security

I am sure that most who have a lab environment in their home also have a way of remotely accessing it – either from at work, with friends or family, vacation, etc. The problem with any remote access into a secure network is that you are quite literally punching a hole into your network from a security sense to allow that to happen.

People seem to have a lot of mixed feelings about allowing Remote Desktop Protocol (RDP) into their home network from the Internet. As a general blanket statement without context, I would completely agree. Opening RDP (port 3389) directly to the Internet without any other security measures in place is asking for trouble. The default RDP port will be constantly brute forced, port scanned, exploited, and the list goes on.

With that said there are steps you can take to have secure remote access to your home network using RDP, SSH, etc.

Many of these concerns can be minimized or eliminated using some of these best practices:

  1. Restrict access using firewalls
  2. Change the listening port for Remote Desktop Protocol
  3. Use two-factor authentication
  4. Use strong passwords
  5. Set an account lockout policy

1 – Restrict access using firewalls

Having a proper firewall and firewall rules in place is critical for protecting your network from outside threats. And I’m not talking about the built-in firewall on your ISP’s provided/rented crappy router/modem – these are a very poor excuse and implementation of a “firewall”- not to mention your ISP normally hard codes back-doors and default logins. I’m talking about a real firewall either physical or virtual, for example; Cisco ASA, Cisco Meraki, Untangle, PF Sense, Sophos UTM/XGUbiquiti, etc. Any of these will give you the tools required to properly firewall your home network. All of these firewalls will require a ‘geek’ to properly setup – keep in mind this article is targeted to home lab hobbyists.

basic_networkTo the right is a good example of a very basic home network with a firewall. Anything before the firewall we treat as untrusted. The firewall is literally the barrier between your network and the big, bad, Internet. There you will define your firewall rules to allow remote access and other functionality (or lack-thereof).

Personally, I use Sophos UTM (see my homelab) with a DNAT (Destination Network Address Translation) rule to redirect the external facing remote access port to a specific server and port on my internal network. This allows me to create a matching condition (For traffic from, Using service/port, Going to) to apply an action (Change the destination to, And the service to) to define what happens when something wants to connect to my network. Using that logic I can (and do) restrict the IP blocks allowed to connect to my remote access port, what times of day, etc.

This allows me to both A; define a custom externally facing port without having to change the port on the server internally, and B; create firewall rules to restrict access even further from specific traffic sources, destinations and services.

The real-world implementation of this will vary based on your choice of firewall, your skills and personal preferences.

2 – Change the listening port for Remote Desktop Protocol

Changing the listening port of RDP is a quick and easy method of implementing security through obscurity. Doing so will help to hide your RDP port from threats who scan networks looking for computers listening on the default Remote Desktop port (TCP 3389).

There are a number of ways to accomplish this. 1 – port redirection on your firewall/router, 2 – modifying the registry keys of the Windows computer locally, or 3 – using a Windows TS Gateway. Choose the method that works best for you.

3 – Use two-factor authentication

Using 2FA (two-factor authentication) is a no brainier these days. Two-factor authentication provides a second layer of security to any type of login, requiring extra information or a physical device to log in, in addition to your password. This protects user logins from remote attacks that may exploit stolen credentials.DuoScreen_740
I use Duo Security Personal edition on my remote RDP access to my home environment. I have configured Duo to only prompt 2FA if the source IP is external. That way I don’t need to use 2FA for local RDP sessions from within my LAN – which would just be annoying. Any time I want to login I just connect, enter my credentials, answer the 2FA prompt on my phone, and I’m in. The Duo Dashboard also has a wide range options, logging, and device fingerprinting. Duo works on a huge number of operating systems and platforms so you can integrate it into, almost, literally any part of your network as you deem fit.

If you are not already using 2FA in your network, start using it! It’s free and extremely easy to setup.

4 – Use strong passwords

While this one may seem like common sense, you would be surprised. A strong password should be at least 8 characters long using a combination of upper and lower case characters – including a mix of both numbers and symbols. Setting an insecure password on anything, let alone a remote entry point to your network could spell disaster.

One of the best ways to ensure that you use unique and strong passwords for systems and websites is to use a password manager. I personally use and recommend Dashlane.

5 – Set an account lockout policy

Brute force attacks are common problem for external facing ports and services. Remember that two-factor authentication only comes into effect once the password is correctly entered and will not prevent a brute force attack. Setting your computer to lock an account for a period of time after a number of incorrect guesses will help prevent attackers from using automated password guessing tools to break into your account.

  • Go to Start–> Programs –>Administrative Tools–> Local Security Policy
  • Under Account Policies –>Account Lockout Policies, set values for all three options.
    • 3 invalid attempts with 3 minute lockout durations are reasonable choices.

Conclusion

Hopefully these tips can help you to increase the security of your home network and remote access methods. If you know what you are doing and if done correctly you can have secure remote entry into your home network. This is not meant to be a be-all-end-all guide as there is no one size fits all for network security. This guide doesn’t even begin to dive into the more complex aspects of network security such as advanced threat protection, intrusion prevention, spoof & protocol protection, and so on.

Have more home lab security tips to share? Post them in the comments below!

New ESXi Server Build – VMH02 Replacement

IMG_0184

This build was originally meant to be a remote ESXi server for my parents place, but I’ve ended up liking this new build so much I’m going to have to keep it for myself. So what I’ll be doing is finishing up this build for my lab and swapping my current 2nd ESXi host (VMH02) to be my MediaPC, and finally re-purposing the MediaPC hardware as an ESXi host for the original plan of the remote lab.

I sort-of figured in the beginning of this remote lab project that I could end up falling in love with the build and deciding to keep it, and well… here we are. I really like the new case (Cooler Master HAF XB EVO ATX) and I’ll be buying another of them for the remote ESXi lab. It’s big/open, lots of fan slots, easy to use and cable manage. That and now that I know how to work with the case properly on the next build it will be super easy to plan out and execute.

ComponentPart NameCost (CAD)
CPUIntel® Core™ i7-950 Processor$50 (used)
MotherboardASUS Rampage III Extreme LGA 1366 Intel X58 SATA 6Gb/s USB 3.0 ATX Intel Motherboard $50 (used)
RAMKingston HyperX Fury Memory Black 16GB 2X8GB DDR3-1866 CL10 - and -
Corsair Vengeance 16GB 2X8GB DDR3-1866
$120
Power SupplyThermaltake TR2 500W Power Supply Cable Management ATX12V V2.3 24PIN With 120mm Fan$50 (have)
CaseCooler Master HAF XB EVO ATX$110
NetworkIntel I350-T4 PCI-Express PCI-E Four RJ45 Gigabit Ports Server Adapter NIC$60
Fans / MiscNZXT Hue 3 RGB Color Changing LED Controller, 2 x 80mm (buy), 1 x 200mm (have), Thermal Compound$50
CPU CoolerCorsair Cooling Hydro Series H60$70
~$560

Once replaced the new VMH02 will be an Intel i7-950 with 32GB of RAM. A small upgrade from the previous i7-920 with 20GB of RAM. I was able to get the used Motherboard, CPU, and 16GB Corsair RAM of RAM (see table above) from a buddy for $120 total.  That alone saved me easily about $600, compared to buying new.

Build Progress:

I’ll another update in the coming days on build progress. 🙂

Storage Refresh 2016 – The Plan

homelab-bottom

The time has come to increase storage capacity in the home lab. I expect that before the end of this year that I will have less than 1TB of free space left on my primary data NAS. That is a problem, and an expensive one at that. At the time of this writing I have 1.66TB of 7.21TB free (77% used). My data growth rate is currently between 3-5% on average per month. That gives me about 2-3 months before I’m in a critical state.

Adding storage to the primary data/media pools means also means adding storage to the backup pools. You won’t catch me without a backup – you only need to be burned by that once before you learn that harsh lesson. Seagate has come out with a 8TB drive meant for backups only which will help with backup capacity. Overall have been pretty skeptical of these 8TB drives. It is strongly advised not to use them in a RAID setup, they use SMR (shingled magnetic recording) that allows the tracks on the platter to be layered on top of each other to increase platter density or tracks per inch (TPI). With that said they seem to be fairly robust. While one could argue that I could (should?) delete some stuff, I strongly disagree. I am a data hoarder. Do you literally throw out all your books after you’re done reading them? Probably not. Same goes with data.

Upgrading the primary NAS means I’ll need to rebuild RAID arrays, use NAS 2 as “swing” storage, move data onto the upgraded NAS 1, rebuild NAS 2, and so on. This will take a couple of days of just moving data around and ensuring I have a backup at all times. During the swing process I am particularly vulnerable to drive failure. Currently my backup NAS 2 is in a JBOD configuration. If any one of the drives fail during this read/write intensive transfer process – game over. For that reason I will be making a second backup onto the 8TB seagate drive, just in case.

The plan is to switch NAS 1 into 5 x 4TB RAID 5, NAS 2 into what NAS 1 is currently (5 x 2TB RAID 5). I’ll then be leveraging my VMH01 (Dell C1100) for the backup pool drives (2 x 8TB, 2 x 2TB in JBOD) served up by a NAS4Free virtual machine. To help wrap my head around what I am doing I like to draw things out on my whiteboard. Here is my “draft” design. Apologies for the chicken scratch.

storagerefresh2016-draft

I’ll be re-purposing an existing 4TB drive in NAS 2 and moving it into the NAS 1 raid pool (hence why only purchasing 4 x 4TB drives as seen below). This saves me the cost of buying another 4TB drive.

I will be using a multi-vendor setup using a mix of Seagate and Western Digital drives. That will make things a little more robust in the long term. Currently I just have desktop rated drives in the primary NAS which, by manufacture guidelines, are only rated for a maximum of 2 in RAID 1/0 and they are also only rated for 8×5 use. The WD Red and Seagate NAS series drives are designed for use in home NAS and servers. They offer a good price to performance ratio, and possess a few features which make them more suitable for RAID arrays such as TLER, higher vibration tolerance (which should result in a longer lifespan), consume less power and are rated for 24/7 use.

western-digital-red-4tb

DriveQuantityCost (CAD)
Seagate ST8000AS0002 8TB 5900RPM 128MB Cache SATA3 Archive Hard Drive OEM - for Backup Data Only2 x $319.99 ea$319.99
Western Digital WD WD40EFRX 4TB Red SATA3 6GB/S Cache 64MB 3.5in Hard Drive2 x $209.99 ea$419.98
Seagate ST4000VN000 4TB 64MB SATA 6GB/S 3.5IN Internal NAS Hard Drive2 x $199.99 ea$399.98
$1,459.94

All said and done I will end up with two large data/media storage pools with 22TB~ of usable combined storage.

A considerably large increase from my existing data capacity of only 7.21TB. The idea being for this to last at least 3+ years. NAS 2 which is currently a JBOD for backups only will now be another usable RAID5 protected data pool. Each NAS backed up to VMH01’s backup storage JBOD.

DeviceCurrent Drive LayoutCurrent CapacityDesired Drive LayoutDesired Capacity
NAS 1 (Thecus N5550)5 x 2TB (RAID 5)7.21TB5 x 4TB (RAID 5)16TB~
NAS 2 (Thecus N5550)1x4TB, 2x2TB, 1x1TB, 1x640GB (JBOD)8.9TB5 x 2TB (RAID 5)7.25TB~
VMH01 (Dell C1100)1 x 1TB1TB2 x 8TB, 2 x 2TB (JBOD)20TB~

I’ll be sure to post updates with pictures on the build and upgrade process when the time comes. For now I’ll be trying to saving up some cash to make this plan come together.