One inch patch cables

Continuing on my slow and steady quest to have an organized networking closet, I recently finished wiring up my patch panel on my 6U “wall-mount” networking rack that’s sitting on top of a pair of IKEA LACK side tables in my office closet. (I used the Cable Matters 1U Cat6 keystone patch panel, which can be found here on Amazon). Previously, I just had all of my apartment’s ethernet cables somehow routed into the closet and terminated at the front of my network switch. It didn’t take long – probably only a few minutes of browsing r/homelab until I realized that this was not nearly the most beautiful way to organize my networking closet. So I set forth on the journey to use a patch panel…

To change from direct ethernet cables to a patch panel, I needed to get:

  • Several keystone connectors. The patch panel I got is a “keystone patch panel” which requires ethernet cables to be terminated with a keystone jack before being inserted into the patch panel. (I got these “slim” Cable Matters ones)
  • Several short ethernet cables which I could use to connect the patch panel to my switch. Sure, I could manually create 16 short cables out of a longer cable, but that sounds like a lot of work. (I ended up buying 4 5-packs of Cable Matters CAT6 cable which can be found on Amazon here)
  • Some way to lengthen my existing ethernet cables without replacing the entire cable. For example, the cable which goes to my desk is the perfect length to reach the front of the switch – but too short to reacharound to the back of the patch panel. Instead of replacing the entire cable, I’d like an easy way to join two cables together. (I ended up getting some Cable Matters “keystone couplers” which can be found here on Amazon)

In all, re-routing the cables didn’t end up being that much work. Using the keystone couplers to lengthen cables was easy. But I’m still not very good at the process of keystone-terminating ethernet cables.. each new termination with its process of stripping and untwisting and punching down took 30 minutes. The whole process took the better part of two days. Despite not testing my cables in the process, I managed to somehow crank out a 100% success rate.

The mess behind my patch panel. The patch panel has “ziptie connectors” that I should probably make use of to clean this up.

Unfortunately, two of my ethernet cables were special “slim” cables which I did not want to strip and attempt to wire into a keystone. Luckily, the keystone couplers I purchased also functioned as a poor-man’s keystone jack, so I just used those for two cables.

After trying to plan in advance and pre-label my patch panel, I was able to connect the 16 keystoned connectors into the rear of the patch panel. I’m not sure if I did it entirely correctly because it still looks a little messy back there – but I did it. I was finally ready to make my patch panel look like all the beautiful r/homelab posts. I used 1 foot ethernet cables to connect the patch panel to my switch. This took some re-working of my already labeled patch panel to get the aesthetics just right. The POE ports on my switch are on the left, so I needed to make sure that all PoE devices were terminated on the left side of the patch panel to avoid crossing patch panels.

The homelab rack after making use of the patch panel – much cleaner!

In all, the rack now looks pretty neat… but there are still things I’d like to improve. For example, the 1 foot patch cables do seem a little too long for the switch ports which are directly below the patch panel. Additionally, this doesn’t seem to actually offer much reparability benefits. For example, if I needed to introduce a new device to my home network, I would need to introduce two new cables instead of just one.. and if I wanted to maintain grouping on the patch panel (living room ports, PoE ports, etc), then I might have to shift over all the patch panel ports to make room. Indeed, the patch panel seems optimized for frequently changing network equipment while the actual ethernet runs remain unchanged – maybe useful in datacenters, but not so much homelabs.

But overall, I’m now proud to show off my network rack and I had fun tweaking. Now all I need is a rackmount UPS, an air conditioning unit dedicated to my networking closet, and to switch my 3 tower servers to rackmount servers. Oh, I also need to upgrade my network to 10 Gig. The fun never ends…

5K

We definitely don’t exercise as much as we should. In fact, when I last went to the doctor he was very insistent that I exercise at least three times a week, drink less alcohol, and drink less coffee. Of course, I haven’t been following his advice as well as I should…

So when my girlfriend came home when day suggesting that we run in a 5K, I was shocked. In general, I have no problem with running (not to brag, but I previously ran a single 5K in college) but she played a sport all her life which used running as the primary form of punishment. She insists that she will never enjoy running because of it.

Anyway – the 5K was scheduled a month from her invitation. Perfect. We’d have plenty of time to train. But of course, as the month went on we skipped almost all the training sessions that we had planned. We ended up running exactly once for one mile.

Come the day of the 5K: We were not ready. My girlfriend had work from 4Am to 8AM and the race was scheduled to start at 9. Also it was supposed to rain. Somehow, this didn’t stop my girlfriend: I woke up at 8:15 to the sound of her barging some screaming “You ready to race?!” I didn’t want to kill that mood, so I got up and got ready. We shared a single apple between us on the drive over.

During the first few minutes of the race, we were miserable. Our legs were already getting sore and I thought that I felt a cramp developing. But running in a group really does help… we couldn’t fall behind! We were encouraged and we kept running.

After the first mile, the people in front of us were far away and the people behind us were far behind. It was just us two. We could take a break or leave the race altogether and no one would know. We could go get pancakes and eggs and no one would say anything.

But something amazing happened. Despite the leg pain and the cramps in the first minutes of the race, we felt no pain now apart from the heavy breathing. We kept running. Just one small break at mile 2, then we ran to the finish. No leg pain. No cramps. 33:18 and 33:19 – my girlfriend beat me by one second.

This may seem like a small feat, but we’re proud of ourselves. What surprises me most is how long it took: just 30 minutes. Surely I’ve spent more than 30 minutes doing anything else: sitting on the toilet doomscrolling, procrastinating actual exercise by working, sitting in traffic. In that time, I could have run a 5K.

So will this become a regular thing? Probably not – but atleast we got over the mountain: a 5K is easy – and we did it.

2023 Homelab Layering

Homelab Overview

My homelab architecture as of October 2022

As we go into 2023, I thought that I would share the layering of my homelab – which I have spent admittedly far too much time on. In general, I spend a lot of time dealing with complex systems during my dayjob. When I’m at home, I try to keep my systems as simple as possible so that I don’t have to spend hours re-learning tooling and debugging when things go wrong. Even if this costs some extra money, it’s worth it because it saves time.

My “homelab” is split into four layers: workstations (actual computers on desks), compute (servers which run applications), storage (server which hosts disks), network (routers and switches).

If I try to get more rigorous about the definitions, I arrive at:

  • Workstations are the client-side interfaces to storage/compute which also have local storage and compute to host games and other local processing (eg photo editing, video editing, word processing, etc). These workstations are backed up to the Storage layer regularly.
  • Compute is the server-side hosting infrastructure. All applications run on “compute”. The compute should probably be using the storage layer as its primary storage, but due to speed considerations (game servers), the compute layer uses its own local storage and backs up regularly to the storage layer.
  • Storage is the server-side storage infrastructure. All storage applications (NFS, SMB) are hosted on storage as well as some other applications which should probably be moved to compute, but are currently on storage due to a lack of research (eg Synology Photos, Synology Drive).
  • Network is the network backbone which connects the workstations/compute/storage and routes to the Internet. My network currently supports 1GBPS.

In my homelab, I’ve decided to split each layer into separate machines just to make my life easier. Workstations are physical computers on desks (eg Minibox or brandon-meshify), Compute is a single machine in my closet, Storage is a combination of a Synology 920+ (for usage) and a Synology 220+ (for onsite backups of the 920+), and Network is a combination of Unifi hardware (Dream Machine Pro, Switch16, 2 APs).

It’s definitely possible to combine this layering – I could imagine a server which performs both compute and storage or maybe a mega-server which performs compute and storage as well as networking. But for my own sanity and for some extra reliability, I’m choosing to host each layer separately.

Dependencies Between Layers

Furthermore, each layer can only depend on the layers below it.

  • The network is the most critical layer and every other layer depends on the network. In cases where the network must depend on other layers, workarounds must be present/documented.
  • Storage depends on the network, but otherwise has no dependencies. If storage is down, we can expect compute and workstations to not operate as expected.
  • Compute depends on storage and network; however, since the compute layer is by definition where applications should be hosted, there are a few instances of applications which lower layers might depend on.
  • Workstations are the highest layer and depend on all other layers in the stack.

Again, I have these rules to keep the architecture simple for my own sanity. If my NAS goes down, for example, I can expect basically everything to break (except the network!). I’m okay with that because creating workarounds is just too complicated (but definitely something I would do in my dayjob).

As I mentioned in the bullet points; there are some exceptions.

  • The compute layer runs pi-hole to perform DNS-level adblocking for the entire network. For this to work, the IP Address of pi-hole is advertised as the primary DNS server when connecting to my home network. Unfortunately, this means that if the compute layer is down, clients won’t be able to resolve IP addresses; I need to manually adjust my router settings and change the advertised DNS Server.
  • The compute layer runs Wireguard to allow me to connect to my network from the outside world. However, if the compute layer is down then Wireguard goes down which means that I won’t be able to repair the compute layer from a remote connection. As a backup, I also have a L2TP VPN enabled on my router. Ideally, my router would be running a Wireguard server since Wireguard is awesome – but the Unifi Dream Machine Pro doesn’t support that natively.
  • My brain depends on the compute layer because that’s where I host wikijs. If the compute layer goes down, my wiki becomes inaccessible (that’s also where I have wiki pages telling me how to fix things). To mitigate this, I use wikijs’s built in functionality to sync articles to disk and I sync them directly to the storage layer. If compute goes down, I should be able to access the articles directly from storage (although probably with broken hyperlinks).

My Fears

Overall, I’m happy with this homelab setup and it’s been serving me well for the past year. I’m a pretty basic homelabber and my homelab isn’t even close to the complex setups I see on YouTube and Reddit. My homelab is running a few game servers (Valheim, Ark, Minecraft), Media Hosting (Jellyfin), and some utilities (pi-hole, wikijs) all under proxmox. This is not nearly as crazy as the folks who have tons of security cameras, the *arr stack to automagically and totally legally grab media on demand, or folks who use their homelab to perform their actual compute-heavy dayjobs.

That said, I’m still worried about the lack of general high-availability in my homelab. For example: if my single UPS for all layers dies during a power outage, if my single compute host dies, if any of my networking equipment die. Any of these issues will knock out my homelab for several days while I wait for replacement hardware to arrive.

But then again, that’s probably okay – having multiple compute nodes or multiple network nodes is a huge added expense with perhaps minimal upsides. After all, I’m not running 24/7 services for customers – it’s just me and my family.

Definitely a work in progress…

Using a Synology NAS: 1 year later

My Synology 920+ and custom-built “compute”

My Long Relationship with Dropbox

For basically as long as I’ve been doing my own software development, I’ve used “the cloud” to store a lot of my data. Specifically, I used Dropbox with the “packrat” option which backed up all of my data to the cloud and even kept track of per-file history. This worked very well – I stored all of my important information in my dropbox folder and when I got a new PC I would just sync from dropbox. When I was away from the PC, I could always access my important documents through the Dropbox App. Dropbox even gave me a reason to avoid learning how to use proper version control because I could just write code directly in Dropbox and let Packrat keep track of the rest.

The list of Dropbox pros is long – but after a decade of using it, I started to run into some problems:

  • Removal of the “public” folder: For the first 5 years of using Dropbox, I made heavy use of the “Public” folder. This folder would allow you to share files with anyone on the internet. I personally used it as a sort of FTP server to host all of my software and updates. Once the public folder was removed, I migrated to a proper FTP server which was much harder to use and diminished the value of Dropbox’s offering.
  • Cost: With Packrat, I was paying $15 per month. In total, I’ve paid ~$1,800 to Dropbox to store my files for the past decade. This is a lot of money – and it’s always been worth it for me to not have to manage things myself, but it’s something to consider.
  • Proliferation of Cloud Services: There are many cloud services available – and all seem to specialize in different things. Google Drive is great for photos and google docs, OneDrive is great for word documents, and Dropbox is great for everything else. Using all of these services meant that my files were never in one place. This isn’t a Dropbox problem – this is more of a me problem – but it was a problem nonetheless.
  • Security: Finally, Dropbox is nice for things that aren’t very private; but I found myself storing passwords and other important information in my Dropbox for lack of a more secure solution. Not good.
The cost of dropbox: I don’t have complete billing history, but I’ve definitely paid more than $1,800 in total

The Synology 920+

The issues with Dropbox nagged at me for a while, but I didn’t do anything about it until Black Friday of 2021 when I purchased a Synology 920+ with 4 Seagate Ironwolf 4TB drives on a whim. I didn’t have a plan – but I had the vague idea that I’d get rid of my Dropbox subscription entirely.

And that’s exactly what I did. The software that comes with the NAS is pretty nice (Synology DSM), so I was able to quickly setup RAID on my drives with single drive fault tolerance and began migrating Google Photos to Synology Photos, my media off of my desktop drive and only the Synology (to be consumed with Jellyfin), and my Google Drive and OneDrive files migrated.

The migration went smoothly, but was basically an archaeological dig through all of my old stuff. I found old screenshots (embarrassing), gamesaves (impressive), and plenty of old schoolwork that I didn’t care about across the 12 harddrives that I had lying around.

I haven’t thought through a proper full review of the 920+, but I’ve been pretty happy with it. The stock fans are a little noisy (fixable, but “voids the warranty”), the stock RAM is not quite enough (fixable, but “voids the warranty” if you buy RAM that’s not Synology branded), and the plastic case leaves a bit to be desired. But functionality wise, I’m happy with it and I’m also happy that I got an “out of the box” software experience and have no need to go digging for other software to manage my photos and files.

Once the migration was complete, I was able to cancel my Dropbox subscription. In all, I paid $1,115 for the hardware which is quite the lump sum (that would have paid for 6 years of Dropbox!) – but I now have 5 times the raw storage and more flexibility to store whatever I want with the peace of mind that it’s not being stored on someone else’s server.

It’s the homelab bug

But my adventure did not stop with buying a Synology. As soon as I got my files switched over to the Synology, I decided I wanted to use the built in Docker features to manage Minecraft, Valheim, and Jellyfin servers – but I soon found that running all 3 of these at once wasn’t a challenge that the 920+ was particularly equipped for…

So I ended up building another system to serve as my “compute” and relegated the Synology to be just storage (I’m hoping to have another post on that “other” system later).

And now that I had these two servers, I needed to buy a fancy network switch to hook them together and then buy a fancy network router to hook to the switch and then fancy APs to enable my wifi…

And even now, I have the aching feeling that running services (eg Photos) from the Synology is not a great idea – instead, I should move the Photos app to the compute layer. And once I do that, I’ll probably need to get a few more servers to ensure high availability for the apps I run.

Oh, what have I done?

The elephant in the room: backups

One of the major selling points of getting a Synology is eliminating the costly monthly bills that you have to pay to cloud providers. The elephant in the room with my new “homelab” is offsite backups which I am currently using Backblaze B2 to achieve. Backblaze B2 requires a monthly subscription, however, so I’m back to where I started: paying a monthly subscription for cloud services.

That said, the monthly subscription is cheaper and I only use it for automated backups (haven’t had to recover yet!).

So what’s the takeaway?

The pessimistic takeaway is that I’ve spent a lot of time and money building Cloud services myself. Besides security, I don’t think there are any actual tangible benefits to doing this – and I wouldn’t recommend it to folks who are looking for storage of a few terabytes.

I’d only recommend this if you have huge amounts of data or if you’re doing it for fun. I guess in my case, I’m having “fun”.

Maximizing My Free Time

Lately, I’ve been feeling pretty stressed about my free time. I’ve taken on so many commitments, both at work and in my personal life, that I don’t seem to have much time for myself. My weekday schedule follows the same pattern of: wake up, do “morning chores” (feed cats, take out their litter, empty the dishwasher, put in a load of laundry, make breakfast, water plants on patio/garden), work from 9-5, exercise, feed the cats, make dinner, watch TV for an hour, go to bed. My weekends are no better. On Saturday, I go grocery shopping at the farmer’s market, tend to our garden for several hours, clean the bathroom or wash our bed sheets depending on which week it is, do any other chores which came up during the week, cook dinner, watch TV, sleep. On Sunday, I: clean the espresso machine, sweep, mop, vacuum, squeeze in an hour of video games, and finish up with an outing that usually lasts the entire day.

By the end of Sunday, I’m exhausted from the long week and disappointed that I didn’t have a solid multi-hour block to sit back, unwind, and do something that I really want to do: like take a really long flight in Flight Simulator, play Minecraft or Valheim, tweak my homelab, or read a book. I usually wake up on Monday anxious about all the chores I need to get done with the goal of getting them done before Sunday so that I will have free time at the week’s end.

I don’t even have kids! I know it will get worse when I do.

Of course when I stumbled upon the “What is your time really worth to you?” quiz from clearerthinking.org on hackernews, I took it immediately. Turns out, my free time is worth a lot to me: on the order of hundreds of dollars an hour. I’m desperate to get just a few more minutes of free time out of my days.

I’m taking the results of my “test” to the extreme – for the past month, every time I have the thought “am I wasting my time?”, I try to find a way to spend money in order to give myself more free time. Here are some examples that I’ve found so far – some of which spend money and some don’t.

  • I registered for Global Entry (and by extension TSA Pre-Check) to save myself time at the airport. This cost $100 and so far, I’ve saved a few hours on our vacation to Banff National Park.
  • I hired cleaners to come bi-weekly to help clean areas which I spend a lot of time on: the stove, bathroom, cabinet doors, and vacuuming cat hair off the couch. This cost $75 per visit and have saved me a few hours so far.
  • In general, I’m trying to look things up on the Internet less. I find that I spend way too long searching for something “perfect” when in reality a single recommendation will do. For example, we recently went on a vacation to Banff National Park and actually used the Visitors Center that we seemed skip on all of our other vacations. We had spent a few hours trying to figure out plans for Canoeing – the visitors center gave us an answer in less than 5 minutes. Easy.
  • If I buy food and water when I’m hungry, even if it means paying a bit more or sacrificing quality, I noticed that I’m happier than if I waited for something better (while starving). Again, this helped us out in our recent Banff vacation.

I’ll continue looking for ways to give myself back some time. There are a few things that I’d like to try that I haven’t been able to yet: hiring a laundry service to pick up my laundry, hiring a personal assistant to help me manage my email inbox, hiring someone to help come up with my weekly meal plans.

I fully realize that several of these are only possible because I’m fortunate enough to have money to spend. The concept of spending money to get back time is still new to me, however, since I grew up in a relatively frugal household.