This June I graduated from NTU, and by then I had been in our lab for three semesters. In the coming years I will stay here and continue my studies for a master's degree. But there's one important task to finish before the next semester begins – the current IT admin in our lab is also graduating, in August! Since we need a new IT admin, I volunteered. This is not only because I love the lab, but also because I have been having a great time setting up my home lab recently and want to manage something on a larger scale.
Overview of the Lab¶
We have more than 10 machines running in our lab. Some of them are in the lab, and the rest are remote in the server room managed by the department. Most of the them are Linux, and lab members can SSH into them and run experiments.
We also have pfSence and PVE. Both are critical services, and other servers cannot work without them. pfSense acts as a firewall that blocks and redirects all the network traffic between the lab and the internet. If it went down, the lab would be isolated from the internet. On PVE, we run FreeIPA as out LDAP server, which the other servers rely on to authenticate users. Without it, no one except the IT admins would be able to log into any server.
That's an overview of the lab. I think it's a decent setup, but as a home lab enthusiast, I want to make it more reliable and resistant to service outages.
Documentation is the Foundation¶
A good rule of thumb is to take notes while setting up systems, so I decided to build a documentation system first. Well, to be honest, that's probably not my real motivation. We had been keeping our notes and docs in a GitHub repository, which worked, but updating them was annoying.
Last September, I proposed that we have a dedicated note system to manage our docs and reduce the friction of updating them. That did not happen, though, because I was very busy at the time and was not an IT admin yet.
After I became the IT admin, the very first service I brought up was Outline, an open-source documentation platform. I learned about it during my internship at Synology, where our team self-hosted it for documentation.
I configured it with a Slack integration, which let users log in with their Slack workspace accounts. I'm very satisfied with it because of its usability and reliability, and I plan to have lab members migrate from the GitHub repository to it once the semester begins.
Keep More Eyes on the Machines¶
Since I'm the only IT admin in the lab, it's impossible for me to keep an eye on every machine. A good monitoring service like Grafana is therefore indispensable. I first heard of it at this year's SITCON, when I thought it was a cool tool but not something I would need. Now, however, was the perfect chance to give it a try!
I deployed Node Exporters on every lab machine to track resource usage, especially hard disk, and online status. I set up notifications that are sent to Slack through a webhook, so that I can fix problems as soon as they appear.
Reclaiming Public IPs¶
Originally, every lab server had its own public IP. These addresses are assigned by the department, so the number we can use is limited. When I became the IT admin, I was asked to set up a new server, but we had already run out of public IPs. To reclaim some, I created two VMs on PVE as SSH jump hosts. By doing so, only the jump hosts need public IPs, while those for other servers can be released.
3-2-1 Backup¶
For me, hosting services for others without backups is unacceptable. As I mentioned, our important services run on PVE, but they were not being backed up regularly, which was extremely dangerous. I installed PBS (Proxmox Backup Server) on an old PC in the lab with a few HDDs mounted, so the PVE can back up its VMs or containers to it.
This is not a proper 3-2-1 backup setup yet, since there are only 2 copies and none of them is off-site. If I have time in the future, maybe I'll deploy a remote PBS or an S3 storage.
Upgrading the EOL PVE¶
While doing all of these, I ran into two PVE crashes. On top of that, Proxmox VE 8 will reach EOF at the end of this August. So I thought it was good time to upgrade PVE to v9. Before upgrading the production machine (call it pve), I did a fresh PVE 9 installation on another PC (pve-2). First, I stopped everything on pve, backed it up to PBS, and restored it on pve-2. After that, I upgraded pve to PVE 9 by following the steps in the PVE documentation. Finally, I stopped everything on pve-2 and started it again on pve, which completed the upgrade. The whole process took me only 3 or 4 hours, much less time than I had expected.
Thoughts¶
I learned a lot about systemd, Docker, and backups by configuring my own home lab, and after working on a real lab I have gained more experience in firewalls, DNS, and Proxmox. I believe that setting up home lab services is a great way to learn Linux and IT skills, and I encourage everyone to try buliding their own home lab. It's a lot of fun!