Wanted to share my setup so far, in part to get some feedback, in part to share results of my own research.
Hardware
The base is an old Dell laptop with 8GB of RAM, nothing special.
For network I got cheap (~30 euro) Cudy router compatible with OpenWRT. I looked at MikroTik but it was more expensive and the setup looked more complex. OpenWRT was very easy to install and fairly easy to set up. I really appreciate the firewall, I was able to easily cut off my smart TV form the internet for example. Setting up port forwarding was also easy.
For storage I got Icy Box USB RAID, set it up to RAID 1. It was the cheapest solution I found and it works fine after a bit of fiddling (issues with the fan but it’s a long story, can give more details if someone’s interested because otherwise I’m happy with it). I use it for backups for now and plan on using it for slow storage in the future. I also got 5TB USB drive for media (arr). I don’t care if I lose it so no backup or RAID here.
No UPS so far. I’m planning on installing solar planes with battery which should protect me from power cuts.
Software
Proxmox as a base. I hesitated if I really need it as I was planning to setup 3 VMs only but in the end the ease of backups and storage management convinced me to use it. Works great, no issues here.
VMs based on Debian.
I’m using cosmos cloud to manage my apps (https://cosmos-cloud.io/). I compared couple of different solutions and this one had the biggest library of supported apps and uses docker (I like the additional security provided by containers). It works great so far and has all the features I needed.
I chose netbird for VPN and I’m not happy with it. The Android app has serious issue with battery usage. It was reported long time ago and it’s still not fixed. That’s really sad because otherwise the app is great. I’m trying to switch to netmaker now. Why not tailscale? I don’t want to tie myself to a closed source app like that.
Backups
VMs backed up by Proxmox daily, stored on the RAID and synced to external server (VPS with NFS drive).
Network and VMs
One internal server accessible only thorough VPN. It hosts *arr stack and bitwarden.
One external server accessible through port forwarding in OpenWRT and nginx with fail2ban. It will host forgejo, my webiste and so on.
One exit node VM running nord VPN.
Phone with always connected VPN routing everything through the exit node.
Bitwarden with self-signed cert imported on phone.
Monitoring
Simple monit scripts pinging individual servers, checking VPN status and status of hard drives.
I’ve been running it for couple of months and so far everything is working great. I want to setup the external server next, test the backups and switch to self-hosted netmaker for VPN.
Anything else I should do or anything I should stop doing?
My setup is pretty similar. Could you help me with whether I should use proxmox?
Currently have everything on my gaming pc, but moving to it’s own dedicated pc. I have about 15 services running, all docker compose.
Debating between proxmox or debian. And was leaning towards Debian as I think I’d just have one VM of Debian with all my docker stuff, and I figured it being a VM would just add extra complications.
Do you think proxmox would be worth it in my case, or am I completely wrong in my thinking of putting all my dockers in one vm, and that’s not how it should be done?
In my case I wanted to separate my services between the ones accessible from the internet (git. lemmy) and not (bitwarden, *arr). To do this I need 3-4 VMs. I went with proxmox for the backups: you just define retention policy and that’s it. Promox drops backups of VMs automatically keeping the right amount of old versions. If you don’t want to split your services and you’re fine with just one VM I think you can easily go with pure debian. I did have some issues with proxmox (like failing network card, it required some fiddling) so it’s not “free”. There’s some extra work to get the benefits.
And I think having everything in one VM is also fine. I’m only splitting it because I want to have some things accessible from the internet. If not I would just use docker for everything.
Thanks. And, yeah I don’t have anything that needs to be accessible via the internet except plex, and plex handles that. Everything else, I’m fine with tailscale.
Here’s a tip: it’s one less firewall to manage, if you make your server be the router, and have your Cudy do only WiFi in dumb AP mode. Or, if the server is in a good enough location, install ´haveged´ and have it serve WiFi too.
Is there a particular reason you’re not selfhosting Wireguard, instead relying on a commercial provider? I’m running this thing on my phone, via F-Droid: https://apt.izzysoft.de/fdroid/index/apk/com.zaneschepke.wireguardautotunnel (it eats battery if you enable keepalive, but you normally don’t need keepalive, so leave it unset).
I tried just using WIreguard and I struggled with with setup. When it fails to connect it’s really hard to figure out what the issue is (port forwarding on one of the routers? firewall on the server? something else?). WIth my netbird setup I’m also relying on DNS, groups and exit node setup. I think it’s just easier with a clear dashboard like the commercial tool have. Netbird is also free and I’m planning on selfhosting netmaker.
I like these posts, I feel like there are many levels to any self hosted setup and getting an idea of what everyone is doing is fun.
I do too. I like people just sharing how their set up is, how it operates, what they do, how they did it. Because there are soooo many ways do do just a single thing, and I like to cross compare what I have to what someone else has done and maybe pick up some tips and pointers most of the time.
We should have a My Set up time for these types. Like open mic at a blues joint. Just get up tell all, show it off, give all the details, the snags you have come up one, the things that really work well. The whole shebang.
Cosmos Cloud over the alternatives is a solid pick—the library size matters more than people admit when you’re running lean on RAM. The post cuts off mid-sentence on that comparison, but I’m guessing you evaluated Portainer, Dockge, or maybe Caprover before landing on Cosmos? Each has different pain points at the 3-VM scale. The reason I ask: app management gets friction-y fast once you’re managing backups and storage tiers and network rules through separate tools. If you want to see how others have structured similar setups—particularly around Cosmos + Proxmox interop—there’s a rundown here https://cxgo.ai/l/QZpbM90 that might save you a rebuild later. But mostly curious what made the library size the deciding factor for you.
Looks like you’ve got it all down on a note. Good job. Tailscale is half n’a halfa (as they say around these parts). Some of it is opensource, some of it is proprietary.
Looks good.
Only thing I would caution on is all of that USB storage. USB isn’t exactly known for it reliability when used long term for storage devices. Given the current market, however, It will do. When you can afford to, I would move the server into a tower with the storage on SATA.
Just to note - or a standalone NAS.
Personally I have two NAS and one older PC ive turned into a NAS, both options work just fine, becomes a question on price/availability and a motherboard that supports the drive quantity you have/want to expand to.
