Keep Proxmox Backup Server Low Dependency

I recently migrated my Proxmox installation from a single drive to a dual drive in ZFS Raid1. The actual process of switching from a single drive to a dual drive was pretty easy. I backed up all of my VMs, installed the new drive, and then reinstalled Proxmox (allowing it to reformat my drives along the way).

The hitch came when restoring from backups. Two things went wrong:

  1. I didn’t have backups for Proxmox Backup Server, so I needed to install it from scratch in order to recover my other VMs.
  2. When I tried to bring up my Proxmox Backup Server, I had difficulty mounting my NAS (where the backups are actually stored) because I only know the NAS by hostname. Hostnames are resolved via my DNS Server (Pihole) which runs in a VM on my proxmox host.

As you can imagine, the fix was pretty simple – installing Proxmox Backup Server from scratch is quite easy and once I figured out the IP of my NAS, I just had to mount it and all was well. But I could make it easier for next time…

Backing up Proxmox Backup Server

The reason I didn’t have backups for Proxmox Backup Server was to avoid circular dependencies. All of VMs were set to backup nightly using Proxmox Backup Server.. but I didn’t want to backup Proxmox Backup Server to itself, so I excluded it from the backup job.

To resolve this for next time, I created a separate backup job which backs up Proxmox Backup Server directly to my NAS. This backup job is scheduled to run after the backup job for the other VMs and follows the same retention policy.

Mounting NAS using IP

I like my home network to be “IP-less”. That is, my router assigns IPs and at the same time advertises their hostnames as local DNS entries. That way, I don’t have to remember IP addresses. However, my router advertises Pi-hole (running in Proxmox) as the primary DNS Server and Pi-Hole uses the router as its upstream DNS.. so if Pihole is down, I can’t use hostnames anymore.

To resolve this, I setup my router to give my NAS a static IP (For some reason, I feel weird about making the NAS itself request a static IP) and then I setup Proxmox Backup Server to mount that IP instead of mounting my NAS by hostname.

In the future, I should be able to restore Proxmox Backup Server from backups hosted on my NAS and it won’t have any problems starting up and mounting my NAS.

The Final Flow

If I need to reinstall Proxmox again, the flow should be a lot less problematic.

  1. Ensure that my NAS is up and running: it has all the backups.
  2. Restore Proxmox Backup Server from the backups hosted directly on my NAS. This should work flawlessly because it is configured to mount my NAS via a static IP.
  3. Add “Proxmox Backup Server” as a new storage in Proxmox. This will allow me to restore my other VMs.
  4. Restore my other VMs.
  5. Create new backup schedules as described above.

Configuring a homelab reverse proxy

As you’ve no doubt noticed from my past 3 posts (all of which are about my “homelab”), I’ve definitely caught the homelab bug. My original goal was simply to get rid of my monthly payments to Dropbox and have some alternative to Google Photos – but this has turned into a hobby of sorts. Now, I’m kinda obsessed with getting all of my regularly running services and game servers well integrated into my homelab environment.

Last post, I talked about organizing the actual hardware into a rack and using a patch panel to route everything. Since then, I actually ended up moving the Proxmox box from an InWin Chopin case to the 2U Rosewill RSV-Z2600U case. The migration was very easy (the new case is HUGE compared to the old one), but required me getting a new power supply and 3 Noctua 80mm fans. So far, I’m very happy with the thermals and noise levels. Most importantly, it looks sleek sitting inside my rack – although now I’ve realized that my rack is a short-depth rack (14 inches) which doesn’t fit the 16 inch 2U case. So I’ll have to do something about that later…

On the software side of things, I’m slowly working towards moving to the ideal homelab. I’ve setup some goals for myself:

  1. Configure an internal reverse proxy so that I don’t have to remember port numbers and so that I don’t get SSL warnings everywhere.
  2. Finally split my network into four: a local network, a guest network, an IoT network, and a “DMZ” network which will run services exposed externally.
  3. Experiment with a “jumphost” – a remote VPS which is connected to the DMZ network via VPN. All connections to the DMZ hosts must then be through the jumphost and the only exposed ports on my home network should be a VPN for both the DMZ network and my local network. For now, this will only be used for game servers.
  4. Experiment with some way to bridge my homelab and my already deployed VPS (this blog and a few other websites).

The journey begins with the internal reverse proxy.

Software Choice: Nginx Proxy Manager

Apparently there are a few reverse proxies to choose from these days. In addition to the classic Apache and Nginx, I also found information about Nginx Proxy Manager, Traefik, and Caddy. I recently got experience with Nginx when I moved this blog from CentOS/Sentora to ArchLinux/Nginx, so I decided that I would stick with the Nginx stack. In addition, Nginx also recently added support for reverse-proxying TCP/UDP while most other options only had HTTP support. This would be useful for gameservers.

So the choice came down to using vanilla Nginx and using Nginx Proxy Manager.

  • Nginx: I already know how to set it up, but I often forget the config file syntax and have to Google about how to use it. Luckily, I don’t plan on doing anything fancy (just subdomain –> hostname/port mapping) so this shouldn’t be too hard. Also I can easily backup the configs.
  • Nginx Proxy Manager: Basically nginx (fast), but with an easy to understand UI. I wouldn’t have to Google anything, but I risk the UI creating janky nginx configs or not supporting an nginx feature that I want later down the line. For backups, I can just host this on a Proxmox container and just backup the container image.

Ultimately, the easy-to-understand UI in Nginx proxy manager won me over. It also supports TCP/UDP (Which nginx calls “streams”) which was a plus.

The “Proxy Hosts” page from the official Nginx Proxy Manager website

Nginx Proxy Manager Setup

Setting up Nginx Proxy Manager was easy-peasy using Docker Compose. I’m definitely not the biggest Docker power-user, but I like using Docker Compose for easy-peasy spinup. In Proxmox, I have a Debian template image with docker/dockercompose pre-installed that I use for all my VMs. I clone the template, fill out docker-compose.yml, and I’m off. It’s probably not great that I’m only running a single docker image in each container, but for now it’s totally fine.

So – I created a new “npm” container and copy/pasted the Docker Compose config from the website. It worked like a charm:

services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ./dockerdata/data:/data
      - ./dockerdata/letsencrypt:/etc/letsencrypt

I started setting up “Proxy Hosts” for services which had ports that I could not remember. The first was proxmox (who can remember port 8006? not me) – so I created the “proxmox.[internaldomain.com]” proxy host and redirected it to the host “proxmox.[internaldomain.com]:8006”.

You might see the problem here… in my homelab network, I votwed to never use IPs directly because I would forget them. Instead, I relied on my Unifi Dream Machine’s internal DNS settings which automatically created entries for [Hostname].[InternalNetwork.com], where InternalNetwork.com was a domain name that I populated (which I also own publicly). I was already navigating to proxmox.[InternalNetwork.com]:8006 to access the Proxmox UI.. so if the hostname already took the “proxmox” name, what would I name the Proxy Host?

I spent an emabrrasingly long time thinking about this and googling to see what other people had done. There were three primary options I was considering:

  1. Just avoid collisions. Keep the Proxy Hosts and actual hostnames on the same domain, but keep a mental note to avoid collisions. I hated this idea because I’m bad at remembering things and I also feared adding some new host down the line that collided with a proxy host.
  2. Create a new subdomain for the proxy (something like proxy.[InternalDomain.com]). I also wasn’t a fan of this idea.. the whole point of creating the proxy was so I could remember less and I didn’t want to remember that everything served by the proxy had to be prefixed with “.proxy”.
  3. Move all hostnames to a different subdomain and use the top-level domain only for the proxy. This is what I ultimately went with.

I realized that eventually I wanted to split out my network into four networks (guest, iot, dmz, and local) – so now would be a perfect time to create a new subdomain for each network starting with local.[InternalDomain.com] for all hosts on my local network. Then, the proxy could have toplevel service.[InternalDomain.com] records and delegate them to the hosts in the local network.

Of course, this took some patience as I had to change a bunch of config files which referred to hosts: hostname.[InternalDomain.com] -> hostname.local.[InternalDomain.com]. But it worked.

Since I use pihole for my internal DNS server (adblocking), I setup my proxy A/CNAME records there:

  • I added an A record proxy.[InternalDomain.com] -> npm.local.[InternalDomain.com].
  • For every service served by the proxy, I added a CNAME service.[InternalDomain.com] -> proxy.[InternalDomain.com].

I setup all of my HTTP internal services this way: Synology Drive, Synology Photos, Unifi Controller, Jellyfin, and Proxmox.

Actually, setting up Proxmox without SSL didn’t quite work. Apparently the Proxmox internals only work over SSL, so I needed to use a special config in Nginx Proxy Manager to redirect all Proxmox to SSL. To do so, I needed to figure out exactly how the UI created “custom configs” and where the “custom configs” were included. Here’s what I used (note the mismatching braces which match with braces in the top-level config where the custom config is included):

rewrite ^(.*) https://$host$1 permanent;
}
server {
  listen 443;
  proxy_redirect off;
  server_name proxmox.[InternalDomain.com];
location / {
                proxy_pass https://homelab.[InternalDomain.com]:8006;
                proxy_buffering off;
                client_max_body_size 0;
                proxy_connect_timeout  3600s;
                proxy_read_timeout  3600s;
                proxy_send_timeout  3600s;
                send_timeout  3600s;
        # Enable proxy websockets for the noVNC console to work
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
        }

Finally, SSL

As a final step, I enabled SSL. This was fairly straightforward: I used the built-in option for creating a new wildcard certificate for *.[InternalDomain.com]. Since my hosts are not publicly accessible, I had to use a DNS challenge for this – but no problem.

I then used the UI to “Force SSL” onto my wildcard cert for all proxy hosts. Viola – no more Chrome/Firefox warnings about insecure connections. Once I did this I was also able to remove the gross custom Proxmox config noted above.

Conclusion

After using this setup for about a week, I’m pretty happy. The quality of life improvements are tremendous… I’ve saved atleast a few seconds off every day. Priceless. (When will those seconds surpass the time I spent setting up the reverse proxy? Probably never…)

I still have some doubts about the setup. For example, having all of my internal services rely on a single host for routing isn’t ideal… but if Nginx Proxy Manager is ever having problems, I could always fall back to the hostname and port.

After all, this is a homelab… I don’t want to spend too much time thinking about 24/7 uptime. That’s for my dayjob.

One inch patch cables

Continuing on my slow and steady quest to have an organized networking closet, I recently finished wiring up my patch panel on my 6U “wall-mount” networking rack that’s sitting on top of a pair of IKEA LACK side tables in my office closet. (I used the Cable Matters 1U Cat6 keystone patch panel, which can be found here on Amazon). Previously, I just had all of my apartment’s ethernet cables somehow routed into the closet and terminated at the front of my network switch. It didn’t take long – probably only a few minutes of browsing r/homelab until I realized that this was not nearly the most beautiful way to organize my networking closet. So I set forth on the journey to use a patch panel…

To change from direct ethernet cables to a patch panel, I needed to get:

  • Several keystone connectors. The patch panel I got is a “keystone patch panel” which requires ethernet cables to be terminated with a keystone jack before being inserted into the patch panel. (I got these “slim” Cable Matters ones)
  • Several short ethernet cables which I could use to connect the patch panel to my switch. Sure, I could manually create 16 short cables out of a longer cable, but that sounds like a lot of work. (I ended up buying 4 5-packs of Cable Matters CAT6 cable which can be found on Amazon here)
  • Some way to lengthen my existing ethernet cables without replacing the entire cable. For example, the cable which goes to my desk is the perfect length to reach the front of the switch – but too short to reacharound to the back of the patch panel. Instead of replacing the entire cable, I’d like an easy way to join two cables together. (I ended up getting some Cable Matters “keystone couplers” which can be found here on Amazon)

In all, re-routing the cables didn’t end up being that much work. Using the keystone couplers to lengthen cables was easy. But I’m still not very good at the process of keystone-terminating ethernet cables.. each new termination with its process of stripping and untwisting and punching down took 30 minutes. The whole process took the better part of two days. Despite not testing my cables in the process, I managed to somehow crank out a 100% success rate.

The mess behind my patch panel. The patch panel has “ziptie connectors” that I should probably make use of to clean this up.

Unfortunately, two of my ethernet cables were special “slim” cables which I did not want to strip and attempt to wire into a keystone. Luckily, the keystone couplers I purchased also functioned as a poor-man’s keystone jack, so I just used those for two cables.

After trying to plan in advance and pre-label my patch panel, I was able to connect the 16 keystoned connectors into the rear of the patch panel. I’m not sure if I did it entirely correctly because it still looks a little messy back there – but I did it. I was finally ready to make my patch panel look like all the beautiful r/homelab posts. I used 1 foot ethernet cables to connect the patch panel to my switch. This took some re-working of my already labeled patch panel to get the aesthetics just right. The POE ports on my switch are on the left, so I needed to make sure that all PoE devices were terminated on the left side of the patch panel to avoid crossing patch panels.

The homelab rack after making use of the patch panel – much cleaner!

In all, the rack now looks pretty neat… but there are still things I’d like to improve. For example, the 1 foot patch cables do seem a little too long for the switch ports which are directly below the patch panel. Additionally, this doesn’t seem to actually offer much reparability benefits. For example, if I needed to introduce a new device to my home network, I would need to introduce two new cables instead of just one.. and if I wanted to maintain grouping on the patch panel (living room ports, PoE ports, etc), then I might have to shift over all the patch panel ports to make room. Indeed, the patch panel seems optimized for frequently changing network equipment while the actual ethernet runs remain unchanged – maybe useful in datacenters, but not so much homelabs.

But overall, I’m now proud to show off my network rack and I had fun tweaking. Now all I need is a rackmount UPS, an air conditioning unit dedicated to my networking closet, and to switch my 3 tower servers to rackmount servers. Oh, I also need to upgrade my network to 10 Gig. The fun never ends…

5K

We definitely don’t exercise as much as we should. In fact, when I last went to the doctor he was very insistent that I exercise at least three times a week, drink less alcohol, and drink less coffee. Of course, I haven’t been following his advice as well as I should…

So when my girlfriend came home when day suggesting that we run in a 5K, I was shocked. In general, I have no problem with running (not to brag, but I previously ran a single 5K in college) but she played a sport all her life which used running as the primary form of punishment. She insists that she will never enjoy running because of it.

Anyway – the 5K was scheduled a month from her invitation. Perfect. We’d have plenty of time to train. But of course, as the month went on we skipped almost all the training sessions that we had planned. We ended up running exactly once for one mile.

Come the day of the 5K: We were not ready. My girlfriend had work from 4Am to 8AM and the race was scheduled to start at 9. Also it was supposed to rain. Somehow, this didn’t stop my girlfriend: I woke up at 8:15 to the sound of her barging some screaming “You ready to race?!” I didn’t want to kill that mood, so I got up and got ready. We shared a single apple between us on the drive over.

During the first few minutes of the race, we were miserable. Our legs were already getting sore and I thought that I felt a cramp developing. But running in a group really does help… we couldn’t fall behind! We were encouraged and we kept running.

After the first mile, the people in front of us were far away and the people behind us were far behind. It was just us two. We could take a break or leave the race altogether and no one would know. We could go get pancakes and eggs and no one would say anything.

But something amazing happened. Despite the leg pain and the cramps in the first minutes of the race, we felt no pain now apart from the heavy breathing. We kept running. Just one small break at mile 2, then we ran to the finish. No leg pain. No cramps. 33:18 and 33:19 – my girlfriend beat me by one second.

This may seem like a small feat, but we’re proud of ourselves. What surprises me most is how long it took: just 30 minutes. Surely I’ve spent more than 30 minutes doing anything else: sitting on the toilet doomscrolling, procrastinating actual exercise by working, sitting in traffic. In that time, I could have run a 5K.

So will this become a regular thing? Probably not – but atleast we got over the mountain: a 5K is easy – and we did it.

2023 Homelab Layering

Homelab Overview

My homelab architecture as of October 2022

As we go into 2023, I thought that I would share the layering of my homelab – which I have spent admittedly far too much time on. In general, I spend a lot of time dealing with complex systems during my dayjob. When I’m at home, I try to keep my systems as simple as possible so that I don’t have to spend hours re-learning tooling and debugging when things go wrong. Even if this costs some extra money, it’s worth it because it saves time.

My “homelab” is split into four layers: workstations (actual computers on desks), compute (servers which run applications), storage (server which hosts disks), network (routers and switches).

If I try to get more rigorous about the definitions, I arrive at:

  • Workstations are the client-side interfaces to storage/compute which also have local storage and compute to host games and other local processing (eg photo editing, video editing, word processing, etc). These workstations are backed up to the Storage layer regularly.
  • Compute is the server-side hosting infrastructure. All applications run on “compute”. The compute should probably be using the storage layer as its primary storage, but due to speed considerations (game servers), the compute layer uses its own local storage and backs up regularly to the storage layer.
  • Storage is the server-side storage infrastructure. All storage applications (NFS, SMB) are hosted on storage as well as some other applications which should probably be moved to compute, but are currently on storage due to a lack of research (eg Synology Photos, Synology Drive).
  • Network is the network backbone which connects the workstations/compute/storage and routes to the Internet. My network currently supports 1GBPS.

In my homelab, I’ve decided to split each layer into separate machines just to make my life easier. Workstations are physical computers on desks (eg Minibox or brandon-meshify), Compute is a single machine in my closet, Storage is a combination of a Synology 920+ (for usage) and a Synology 220+ (for onsite backups of the 920+), and Network is a combination of Unifi hardware (Dream Machine Pro, Switch16, 2 APs).

It’s definitely possible to combine this layering – I could imagine a server which performs both compute and storage or maybe a mega-server which performs compute and storage as well as networking. But for my own sanity and for some extra reliability, I’m choosing to host each layer separately.

Dependencies Between Layers

Furthermore, each layer can only depend on the layers below it.

  • The network is the most critical layer and every other layer depends on the network. In cases where the network must depend on other layers, workarounds must be present/documented.
  • Storage depends on the network, but otherwise has no dependencies. If storage is down, we can expect compute and workstations to not operate as expected.
  • Compute depends on storage and network; however, since the compute layer is by definition where applications should be hosted, there are a few instances of applications which lower layers might depend on.
  • Workstations are the highest layer and depend on all other layers in the stack.

Again, I have these rules to keep the architecture simple for my own sanity. If my NAS goes down, for example, I can expect basically everything to break (except the network!). I’m okay with that because creating workarounds is just too complicated (but definitely something I would do in my dayjob).

As I mentioned in the bullet points; there are some exceptions.

  • The compute layer runs pi-hole to perform DNS-level adblocking for the entire network. For this to work, the IP Address of pi-hole is advertised as the primary DNS server when connecting to my home network. Unfortunately, this means that if the compute layer is down, clients won’t be able to resolve IP addresses; I need to manually adjust my router settings and change the advertised DNS Server.
  • The compute layer runs Wireguard to allow me to connect to my network from the outside world. However, if the compute layer is down then Wireguard goes down which means that I won’t be able to repair the compute layer from a remote connection. As a backup, I also have a L2TP VPN enabled on my router. Ideally, my router would be running a Wireguard server since Wireguard is awesome – but the Unifi Dream Machine Pro doesn’t support that natively.
  • My brain depends on the compute layer because that’s where I host wikijs. If the compute layer goes down, my wiki becomes inaccessible (that’s also where I have wiki pages telling me how to fix things). To mitigate this, I use wikijs’s built in functionality to sync articles to disk and I sync them directly to the storage layer. If compute goes down, I should be able to access the articles directly from storage (although probably with broken hyperlinks).

My Fears

Overall, I’m happy with this homelab setup and it’s been serving me well for the past year. I’m a pretty basic homelabber and my homelab isn’t even close to the complex setups I see on YouTube and Reddit. My homelab is running a few game servers (Valheim, Ark, Minecraft), Media Hosting (Jellyfin), and some utilities (pi-hole, wikijs) all under proxmox. This is not nearly as crazy as the folks who have tons of security cameras, the *arr stack to automagically and totally legally grab media on demand, or folks who use their homelab to perform their actual compute-heavy dayjobs.

That said, I’m still worried about the lack of general high-availability in my homelab. For example: if my single UPS for all layers dies during a power outage, if my single compute host dies, if any of my networking equipment die. Any of these issues will knock out my homelab for several days while I wait for replacement hardware to arrive.

But then again, that’s probably okay – having multiple compute nodes or multiple network nodes is a huge added expense with perhaps minimal upsides. After all, I’m not running 24/7 services for customers – it’s just me and my family.

Definitely a work in progress…