You must log in or # to comment.
That’s not a homelab, that’s a home server.
How is the kubernetes (k3s/rke2) migration coming along?
No try migrating all your docker containers to podman.
Just did that last weekend. Nothing to do anymore. 😢
Did you do Quadlets?
Yes of course. Had to spend a couple of hours fixing permission related issues.
But did you run them as rootful or the intended rootless way.
I had problems getting apps with multiple containers working in quadlets (definitely a knowledge issue on my part, but didn’t feel the time learning it was beneficial, but will probably revisit during kubernetes learning) so went back to podman with docker compose.
I think it’s kinda better using quadlets, because I wrote some custom scripts, and quadlets made the process better. But podman compose is probably file too.
And then migrate all your podman containers to proxmox
Don’t encourage me.
And then try turning on SELinux!
I set my homelab up on Bazzite immutable with podman and SELinux. It took a while to work everything out and have it boot up into a valid state hahaha
Any reason you chose Bazzite for your homelab distro? First I’ve heard of someone doing that!
Wouldn’t an immutable OS be overall a pretty good idea for a stable server?
At the start I just wanted a desktop machine that runs Steam through sunshine/moonlight so hardware support and gaming stuff such was very important.
My homelab used to run on my laptop when it could all fit within a couple 100s of GB and I was the only user but moving it was tricky. Since I’m a programmer I’m not afraid of this stuff so I just spent the hours to figure out one problem at a time.
I ended up figuring out adding HDD whitelist in SELinux, make it accessible in podman, manually edit fstab because tools didn’t work, systemd service for startup, logging in automatically where I already forgot everything and would have not had to do any of this on a bog standard Ubuntu server.
Respect! I too often take it for granted that it’s a privilege for my gaming rig and my homelab server to be separate boxes.
My server is Almalinux, my laptop is Mint, and my gaming rig is Nobara. But if I had to consolidate everything in to one machine, I’d pick Nobara.
I came to the same conclusion, Nobara for would have been best.
It’s not that difficult to get SELinux working with podman quadlets, especially if you run things rootless. I have a kerberized service account for each application I host and my quadlets are configured to run under those. I very rarely encounter applications that simoky can’t be run rootless but I usually can find an adequate alternative. I think right now the only thing that runs as root is one of the talk or collabora containers in my nextcloud stack. No selinux issues either.
I use podman-compose with system accounts and I don’t have a ton of issues. The biggest one is that I can’t seem to get bluetooth and pip working on Home Assistant at the same time. Most of the servers I manage have SELinux and it works fine as long as I use
:z/:Zwith bind mounts.A few years ago, I set up a VPS for my friend’s business; at the time, I didn’t know how to work with SELinux so I just turned it off. I tried to flip it back on, and it somehow bricked the system. We had to restore from a backup. Since then, I’ve been afraid to enable it on my flagship homelab server.
are you sure it really bricked it? when turning it on, on next boot it needs to go over all the files and retag them or something like that, and it can take a significant amount of time
Honestly, I don’t know what happened, but it was unreachable via SSH and the web console. There shouldn’t have been a ton of files to tag since it was an Almalinux system that started with SELinux enabled, and all we added was a container app or two.
that started with SELinux enabled
that does not matter, it needs to go over all of them. I don’t know how long it takes with SSD, but with HDD it can take a half an hour or more, with a mostly base system. and the kernel starts doing this very early, when not even systemd or other processes are running, so no ssh, but web console should have been working to see what its doing
Time to start documenting it!
NEVER1!!!11!!
At 71, I have to document. I started a long time ago. I worked for a mec. contractor long ago, and the rule was: ‘If you didn’t write it down, it didn’t happen.’ That just carried over to everything I do.
Do you write down what you write down on the internet?
As in a blog or wiki? I do not because I am not authoritative. What I know came from reading, doing, screwing it up, ad nauseam. When something finally clicks for me, I write it down because 9 times out of 10, I will need that info later. But my writing would be so full of inaccuracies that it would be embarrassing and possibly lead someone astray.
It’s how cults start!
I’ve started to take a l lot more notes at work I guess there will be a time where I take notes of what month it is!
I guess there will be a time where I take notes of what month it is!
You may jest, but there are times when I can’t remember what I had for breakfast. They say that you never truly forget anything, but that our recall mechanism fades over time. For a myriad of reasons, including age, my recall mechanism is shit.
Offt depends what you had and your version of health. I am hopeful that technology helps when I am that age, only a few years but ai agents seem to be a start. Just need to let go of those big data fears.
Don’t look too closely you can jinx it.
You can always configure your vim further
or learn emacs
Then configure vim using emacs
Let’s tinker around and accidentally break something.
and debug it until you have to reinstall your entire stack from scarch
Scarched arth
GET OUT OF MY HOUSE!
Are you implying it’s possible to debug without having to reinstall from scratch? Preposterous! 😂
Guess this is a good time to test my infrastructure automation.
“Damn, I’ve got this Debian server shit down. I wonder how an opensuse server would work out”
*installs tumbleweed*
True story
My
manperson!
I wish it was stable
I had a drive die yesterday
Time to distro-hop!
Backups. You’re forgetting them.
Pro tip: If you’re using openwrt or other managed network components don’t forget to automatically back those up too. I almost had to reset my openwrt router and having to reconfigure that from scratch sucks.
All of those systems in your homelab…they aren’t all pulling down their updates multiple times over your network link, right? You’re making use of a network-wide cache? For Debian-family systems, something like Apt-Cacher NG?
Oh. You’re not. Well, that’s probably okay. I mean, not everyone can have their environment optimized to minimize network traffic.
I set this up years ago, but then decided it was better to just install different distros on each of my computers. Problem solved?
You can forgejo with a container index enabled, I don’t know if there’s a way to use that as a proxy for downloading containers though.
Buy a UPS and setup a NUT server on the spare raspberry pi you have lying around.
You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can’t be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?
Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.
I built an 8 outlet version of those with relays and wall outlets for… a lot less.
If you do have the smart PSU and power management server you probably also went down the rabbit hole of scripting the power cycling, right? Maybe made that server hardened against power loss disk corruption so it can be run until UPS battery exhaustion.
What if there is a power outage and NUT shuts everything down? Would be nice to have everything brought back up in an orderly way when power returns. Without manual intervention. But keeping you informed via logging and push notifications.
Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.
*furiously adds a new item to the TODO list*
Tal just got the chaotic evil tag today.
Does a $12 Shelly plug count?
if you can cycle your home assistant with the shelly plug whilst your home assistant is down, yes. from experience it’s really quite annoying to have a smart plug switch off HA…
HA is on the same proxmox host as the router. So yeah I can end up locked out. Hasn’t happened yet tho! The relay is on my test machine, it’s always nvidia that crashes there.
An 8 switch relay, old Pi, and 8 hardware store outlets can be had for not much more. I did that and let PiKVM control my outlets directly.
This is the back of my 10" rack before it was cleaned up. Lots of custom work on this that I’ll be posting a page on my site about when complete.
@tal@lemmy.today in case you are interested
You should use Arch, then you can update every 15 minutes 🤭
Do your backups work?
Have you tested your backups recently? Having them complete is one thing, having the data you need for recovery is another. Have you backed up your vm configurations and build scripts?
Go test your latest backup!
Restore is future me’s problem. Fuck that guy :D
Ah, that frission of excitement when you come to restore! Will it work? Does it contain that very important file? Is it up to date? How much will future you hate past you if it isn’t there?
When’s the last time you checked if your backup solution works?
What’s a backup solution…? (I’m only being half sarcastic, I really need to set one up, but it’s not as “fun” as the rest of my homelab, open to suggestions)
No mercy for you, then. ;)
I at least have external backups for important family pics and docs! But yea the homelab itself is severely lacking. If it dies, I get to start from scratch. Been gambling for years that “I’ll get around to a backup solution before it dies”. I wouldn’t bet on me :|
But if my backups actually work then I miss out on the joy of rebuilding everything from scratch and explaining to my wife why non of the lights in the house work anymore.
Carry around a candle in one of those old timey holders like Scrooge Mcduck
Backup? Psh… That’s what the lab is for.
Yesterday! Switched my media server from freebsd to alpine and got the arr stack all set up using the backup zip files
