Author Archives: brandonio21

2023 Homelab Layering

Homelab Overview

My homelab architecture as of October 2022

As we go into 2023, I thought that I would share the layering of my homelab – which I have spent admittedly far too much time on. In general, I spend a lot of time dealing with complex systems during my dayjob. When I’m at home, I try to keep my systems as simple as possible so that I don’t have to spend hours re-learning tooling and debugging when things go wrong. Even if this costs some extra money, it’s worth it because it saves time.

My “homelab” is split into four layers: workstations (actual computers on desks), compute (servers which run applications), storage (server which hosts disks), network (routers and switches).

If I try to get more rigorous about the definitions, I arrive at:

  • Workstations are the client-side interfaces to storage/compute which also have local storage and compute to host games and other local processing (eg photo editing, video editing, word processing, etc). These workstations are backed up to the Storage layer regularly.
  • Compute is the server-side hosting infrastructure. All applications run on “compute”. The compute should probably be using the storage layer as its primary storage, but due to speed considerations (game servers), the compute layer uses its own local storage and backs up regularly to the storage layer.
  • Storage is the server-side storage infrastructure. All storage applications (NFS, SMB) are hosted on storage as well as some other applications which should probably be moved to compute, but are currently on storage due to a lack of research (eg Synology Photos, Synology Drive).
  • Network is the network backbone which connects the workstations/compute/storage and routes to the Internet. My network currently supports 1GBPS.

In my homelab, I’ve decided to split each layer into separate machines just to make my life easier. Workstations are physical computers on desks (eg Minibox or brandon-meshify), Compute is a single machine in my closet, Storage is a combination of a Synology 920+ (for usage) and a Synology 220+ (for onsite backups of the 920+), and Network is a combination of Unifi hardware (Dream Machine Pro, Switch16, 2 APs).

It’s definitely possible to combine this layering – I could imagine a server which performs both compute and storage or maybe a mega-server which performs compute and storage as well as networking. But for my own sanity and for some extra reliability, I’m choosing to host each layer separately.

Dependencies Between Layers

Furthermore, each layer can only depend on the layers below it.

  • The network is the most critical layer and every other layer depends on the network. In cases where the network must depend on other layers, workarounds must be present/documented.
  • Storage depends on the network, but otherwise has no dependencies. If storage is down, we can expect compute and workstations to not operate as expected.
  • Compute depends on storage and network; however, since the compute layer is by definition where applications should be hosted, there are a few instances of applications which lower layers might depend on.
  • Workstations are the highest layer and depend on all other layers in the stack.

Again, I have these rules to keep the architecture simple for my own sanity. If my NAS goes down, for example, I can expect basically everything to break (except the network!). I’m okay with that because creating workarounds is just too complicated (but definitely something I would do in my dayjob).

As I mentioned in the bullet points; there are some exceptions.

  • The compute layer runs pi-hole to perform DNS-level adblocking for the entire network. For this to work, the IP Address of pi-hole is advertised as the primary DNS server when connecting to my home network. Unfortunately, this means that if the compute layer is down, clients won’t be able to resolve IP addresses; I need to manually adjust my router settings and change the advertised DNS Server.
  • The compute layer runs Wireguard to allow me to connect to my network from the outside world. However, if the compute layer is down then Wireguard goes down which means that I won’t be able to repair the compute layer from a remote connection. As a backup, I also have a L2TP VPN enabled on my router. Ideally, my router would be running a Wireguard server since Wireguard is awesome – but the Unifi Dream Machine Pro doesn’t support that natively.
  • My brain depends on the compute layer because that’s where I host wikijs. If the compute layer goes down, my wiki becomes inaccessible (that’s also where I have wiki pages telling me how to fix things). To mitigate this, I use wikijs’s built in functionality to sync articles to disk and I sync them directly to the storage layer. If compute goes down, I should be able to access the articles directly from storage (although probably with broken hyperlinks).

My Fears

Overall, I’m happy with this homelab setup and it’s been serving me well for the past year. I’m a pretty basic homelabber and my homelab isn’t even close to the complex setups I see on YouTube and Reddit. My homelab is running a few game servers (Valheim, Ark, Minecraft), Media Hosting (Jellyfin), and some utilities (pi-hole, wikijs) all under proxmox. This is not nearly as crazy as the folks who have tons of security cameras, the *arr stack to automagically and totally legally grab media on demand, or folks who use their homelab to perform their actual compute-heavy dayjobs.

That said, I’m still worried about the lack of general high-availability in my homelab. For example: if my single UPS for all layers dies during a power outage, if my single compute host dies, if any of my networking equipment die. Any of these issues will knock out my homelab for several days while I wait for replacement hardware to arrive.

But then again, that’s probably okay – having multiple compute nodes or multiple network nodes is a huge added expense with perhaps minimal upsides. After all, I’m not running 24/7 services for customers – it’s just me and my family.

Definitely a work in progress…

Using a Synology NAS: 1 year later

My Synology 920+ and custom-built “compute”

My Long Relationship with Dropbox

For basically as long as I’ve been doing my own software development, I’ve used “the cloud” to store a lot of my data. Specifically, I used Dropbox with the “packrat” option which backed up all of my data to the cloud and even kept track of per-file history. This worked very well – I stored all of my important information in my dropbox folder and when I got a new PC I would just sync from dropbox. When I was away from the PC, I could always access my important documents through the Dropbox App. Dropbox even gave me a reason to avoid learning how to use proper version control because I could just write code directly in Dropbox and let Packrat keep track of the rest.

The list of Dropbox pros is long – but after a decade of using it, I started to run into some problems:

  • Removal of the “public” folder: For the first 5 years of using Dropbox, I made heavy use of the “Public” folder. This folder would allow you to share files with anyone on the internet. I personally used it as a sort of FTP server to host all of my software and updates. Once the public folder was removed, I migrated to a proper FTP server which was much harder to use and diminished the value of Dropbox’s offering.
  • Cost: With Packrat, I was paying $15 per month. In total, I’ve paid ~$1,800 to Dropbox to store my files for the past decade. This is a lot of money – and it’s always been worth it for me to not have to manage things myself, but it’s something to consider.
  • Proliferation of Cloud Services: There are many cloud services available – and all seem to specialize in different things. Google Drive is great for photos and google docs, OneDrive is great for word documents, and Dropbox is great for everything else. Using all of these services meant that my files were never in one place. This isn’t a Dropbox problem – this is more of a me problem – but it was a problem nonetheless.
  • Security: Finally, Dropbox is nice for things that aren’t very private; but I found myself storing passwords and other important information in my Dropbox for lack of a more secure solution. Not good.
The cost of dropbox: I don’t have complete billing history, but I’ve definitely paid more than $1,800 in total

The Synology 920+

The issues with Dropbox nagged at me for a while, but I didn’t do anything about it until Black Friday of 2021 when I purchased a Synology 920+ with 4 Seagate Ironwolf 4TB drives on a whim. I didn’t have a plan – but I had the vague idea that I’d get rid of my Dropbox subscription entirely.

And that’s exactly what I did. The software that comes with the NAS is pretty nice (Synology DSM), so I was able to quickly setup RAID on my drives with single drive fault tolerance and began migrating Google Photos to Synology Photos, my media off of my desktop drive and only the Synology (to be consumed with Jellyfin), and my Google Drive and OneDrive files migrated.

The migration went smoothly, but was basically an archaeological dig through all of my old stuff. I found old screenshots (embarrassing), gamesaves (impressive), and plenty of old schoolwork that I didn’t care about across the 12 harddrives that I had lying around.

I haven’t thought through a proper full review of the 920+, but I’ve been pretty happy with it. The stock fans are a little noisy (fixable, but “voids the warranty”), the stock RAM is not quite enough (fixable, but “voids the warranty” if you buy RAM that’s not Synology branded), and the plastic case leaves a bit to be desired. But functionality wise, I’m happy with it and I’m also happy that I got an “out of the box” software experience and have no need to go digging for other software to manage my photos and files.

Once the migration was complete, I was able to cancel my Dropbox subscription. In all, I paid $1,115 for the hardware which is quite the lump sum (that would have paid for 6 years of Dropbox!) – but I now have 5 times the raw storage and more flexibility to store whatever I want with the peace of mind that it’s not being stored on someone else’s server.

It’s the homelab bug

But my adventure did not stop with buying a Synology. As soon as I got my files switched over to the Synology, I decided I wanted to use the built in Docker features to manage Minecraft, Valheim, and Jellyfin servers – but I soon found that running all 3 of these at once wasn’t a challenge that the 920+ was particularly equipped for…

So I ended up building another system to serve as my “compute” and relegated the Synology to be just storage (I’m hoping to have another post on that “other” system later).

And now that I had these two servers, I needed to buy a fancy network switch to hook them together and then buy a fancy network router to hook to the switch and then fancy APs to enable my wifi…

And even now, I have the aching feeling that running services (eg Photos) from the Synology is not a great idea – instead, I should move the Photos app to the compute layer. And once I do that, I’ll probably need to get a few more servers to ensure high availability for the apps I run.

Oh, what have I done?

The elephant in the room: backups

One of the major selling points of getting a Synology is eliminating the costly monthly bills that you have to pay to cloud providers. The elephant in the room with my new “homelab” is offsite backups which I am currently using Backblaze B2 to achieve. Backblaze B2 requires a monthly subscription, however, so I’m back to where I started: paying a monthly subscription for cloud services.

That said, the monthly subscription is cheaper and I only use it for automated backups (haven’t had to recover yet!).

So what’s the takeaway?

The pessimistic takeaway is that I’ve spent a lot of time and money building Cloud services myself. Besides security, I don’t think there are any actual tangible benefits to doing this – and I wouldn’t recommend it to folks who are looking for storage of a few terabytes.

I’d only recommend this if you have huge amounts of data or if you’re doing it for fun. I guess in my case, I’m having “fun”.

Maximizing My Free Time

Lately, I’ve been feeling pretty stressed about my free time. I’ve taken on so many commitments, both at work and in my personal life, that I don’t seem to have much time for myself. My weekday schedule follows the same pattern of: wake up, do “morning chores” (feed cats, take out their litter, empty the dishwasher, put in a load of laundry, make breakfast, water plants on patio/garden), work from 9-5, exercise, feed the cats, make dinner, watch TV for an hour, go to bed. My weekends are no better. On Saturday, I go grocery shopping at the farmer’s market, tend to our garden for several hours, clean the bathroom or wash our bed sheets depending on which week it is, do any other chores which came up during the week, cook dinner, watch TV, sleep. On Sunday, I: clean the espresso machine, sweep, mop, vacuum, squeeze in an hour of video games, and finish up with an outing that usually lasts the entire day.

By the end of Sunday, I’m exhausted from the long week and disappointed that I didn’t have a solid multi-hour block to sit back, unwind, and do something that I really want to do: like take a really long flight in Flight Simulator, play Minecraft or Valheim, tweak my homelab, or read a book. I usually wake up on Monday anxious about all the chores I need to get done with the goal of getting them done before Sunday so that I will have free time at the week’s end.

I don’t even have kids! I know it will get worse when I do.

Of course when I stumbled upon the “What is your time really worth to you?” quiz from clearerthinking.org on hackernews, I took it immediately. Turns out, my free time is worth a lot to me: on the order of hundreds of dollars an hour. I’m desperate to get just a few more minutes of free time out of my days.

I’m taking the results of my “test” to the extreme – for the past month, every time I have the thought “am I wasting my time?”, I try to find a way to spend money in order to give myself more free time. Here are some examples that I’ve found so far – some of which spend money and some don’t.

  • I registered for Global Entry (and by extension TSA Pre-Check) to save myself time at the airport. This cost $100 and so far, I’ve saved a few hours on our vacation to Banff National Park.
  • I hired cleaners to come bi-weekly to help clean areas which I spend a lot of time on: the stove, bathroom, cabinet doors, and vacuuming cat hair off the couch. This cost $75 per visit and have saved me a few hours so far.
  • In general, I’m trying to look things up on the Internet less. I find that I spend way too long searching for something “perfect” when in reality a single recommendation will do. For example, we recently went on a vacation to Banff National Park and actually used the Visitors Center that we seemed skip on all of our other vacations. We had spent a few hours trying to figure out plans for Canoeing – the visitors center gave us an answer in less than 5 minutes. Easy.
  • If I buy food and water when I’m hungry, even if it means paying a bit more or sacrificing quality, I noticed that I’m happier than if I waited for something better (while starving). Again, this helped us out in our recent Banff vacation.

I’ll continue looking for ways to give myself back some time. There are a few things that I’d like to try that I haven’t been able to yet: hiring a laundry service to pick up my laundry, hiring a personal assistant to help me manage my email inbox, hiring someone to help come up with my weekly meal plans.

I fully realize that several of these are only possible because I’m fortunate enough to have money to spend. The concept of spending money to get back time is still new to me, however, since I grew up in a relatively frugal household.

On Fredericksburg Removing the Slave Auction Block

After the killing of George Floyd, the city of Fredericksburg, Virginia removed the historical slave auction block from the sidewalk on the corner of William St. and Princess Anne St. outside of Hyperion Espresso.

The slave auction block on a street corner in Fredericksburg, VA
By Sarah Stierch – originally posted to Flickr as Slave Auction Block, Fredericksburg, VA, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=10596228

At first, I was surprised by the decision. When I visited Fredericksburg, I saw the auction block and it reminded me of atrocious events in American History and how we overcame them. The blocks placement in the middle of a highly-trafficked sidewalk strengthens the point: Hundreds (maybe even thousands) of slaves were stripped of their humanity and traded hands at this very spot. This was part of everyday life in pre Civil War Fredericksburg. Why would th ecity choose to remove such a good reminder of the past and the great progress we’ve since made?

White Fragility and Tears We Cannot Stop gave me insight: Have we made as much progress as I previously thought? When I first appreciated the auction block, was I viewing the events of history through my white-privilege-tinted glasses? What does the auction block mean for a black person in America today? I imagine their life – struggling everyday with systemic racism and discrimination. Every time they walk by this auction block, they are reminded that only 200 years ago black folk were auctioned off to whites in this very neighborhood. That’s the reason they are treated poorly today. That’s the reason 1 in 3 black folks are sent to jail in their lifetime. That’s the reason that innocent blacks are shot down by police.

Black Americans are forced to remember this every time they walk this street or sit down to drink a cup of coffee. Meanwhile, white folk look at the block and remind themselves of how great things are today.

I didn’t understand the burden that the auction block might impose on Black Americans. I now believe that the city’s choice to remove the block was a good one. The block did not represent great progress or achievement. Instead, it only reminded us of why there is such great racism in American today. Removing it from the streets gives black folk the opportunity to walk around this particular corner without being reminded of the immense historical baggage and systemic racism that black folk face every day in America.

That said, the auction block surely still belongs in a museum where people can see it and be reminded while they are in the right mindset and surrounded by historical context.

The Moment Between Flips

In today’s fast paced world, pancakes are a great way to slow down. Each pancake needs to cook for a minute on each side. During that minute, you have to stay close – you can’t go off and do something else. It’s a great time to think. The mind wanders about this and that as the bubbles rise to the surface of the pancake. It’s a peaceful moment with nothing to do except wait for the next flip.

I cherish these pancake moments. I love to image that I’m living in the country and I’ve just completed the morning chores – nothing to do but think. I think about how nice the weather is and how interesting it is to be alive.

Today, I thought about how much we consume. Why do I feel the urge to browse Amazon for new things? Why isn’t what I have sufficient? I am addicted to that small burst of happiness when the next Amazon package arrives. But like pancakes, there’s no need to rush – slow down and enjoy the current moment.