Back in the day it was nice, apt get update && apt get upgrade and you were done.
But today every tool/service has it’s own way to being installed and updated:
- docker:latest
- docker:v1.2.3
- custom script
- git checkout v1.2.3
- same but with custom migration commands afterwards
- custom commands change from release to release
- expect to do update as a specific user
- update nginx config
- update own default config and service has dependencies on the config changes
- expect new versions of tools
- etc.
I selfhost around 20 services like PieFed, Mastodon, PeerTube, Paperless-ngx, Immich, open-webui, Grafana, etc. And all of them have some dependencies which need to be updated too.
And nowadays you can’t really keep running on an older version especially when it’s internet facing.
So anyway, what are your strategies how to keep sanity while keeping all your self hosted services up to date?
You must log in or # to comment.
I don’t use docker, etc, so for me, if it’s in the normal Arch repos or AUR then I don’t need to think about it until there’s a
.pacnewfile to look atThen, it’s just the odd git pull on literally 2 devices.
All organised by ansible…
(well except the
.pacnew, but I think it’s nice to keep in touch with the packages)I keep it simple, although reading down through the thread, there are some really nice and ingenious ways people accomplish about the same thing, which is totally awesome. I use a WatchTower fork and run it with
--run-once --cleanup. I do this when I feel comfortable that all the early adopters have done all the beta testing for me. Thanks early adopters. So, about 1 a month or so, I update 70 Docker containers. As far as OS updates, I usually hit those when they deploy. I’m running Ubuntu Jammy, so not a lot of breaking changes in updates. I don’t have public facing services, and I am the only user on my network, so I don’t really have to worry too much about that aspect.I just run watchtower in docker. It will watch all your other docker images and update them to latest version automatically if you want.
It works fine but with time, I stopped thinking i need to be on latest version all the time. It really isnt very important.
Just a few of my services are open on the internet, mainly caddy and wireguard.
Heads up that watchtower is no longer maintained. I haven’t yet looked into forks or alternatives.
Kubernetes + helm charts
Wow, that sounds like a nightmare. Here’s my workflow:
nix flake update nixos-rebuild switchThat gives me an atomic, rollbackable update of every service running on the machine.
I run NixOS. Go to the flake file and update channel version.
Renovate + GitOps. Check out https://github.com/onedr0p/cluster-template
If you don’t like Kubernetes, you can get a similar setup with doco-CD. Only limitation is that dococd can’t update itself, but you can use SOPS and Renovate all the same for the other services.
That or Komodo when using docker. Renovate is really good, you always know which version you’re at, you can set it up to auto merge on minor and/or patch level, it shows you the release notes etc.
This tutorial is good: https://nickcunningh.am/blog/how-to-automate-version-updates-for-your-self-hosted-docker-containers-with-gitea-renovate-and-komodo
I guess auto merge isn’t enabled, since there’s no way to check if an update doesn’t break your deployment beforehand, am I right?
You can configure automerge per stack and also if it’s allowed on patch, minor or major upgrades.
Yes, but usually when you use automerge you should have set up a CI to make sure new versions don’t break your software or deployment. How are you supposed to do that in a self-hosting environment?
Everything I run, I deploy and manage with ansible.
When I’m building out the role/playbook for a new service, I make sure to build in any special upgrade tasks it might have and tag them. When it’s time to run infrastructure-wide updates, I can run my single upgrade playbook and pull in the upgrade tasks for everything everywhere - new packages, container images, git releases, and all the service restart steps to load them.
It’s more work at the beginning to set the role/playbook up properly, but it makes maintaining everything so much nicer (which I think is vital to keep it all fun and manageable).
Yeah, For some reason I didn’t think of ansible even though I use it at work regularly. Thanks for pointing it out!
Just a word of caution…
I try to upgrade 1 (of a similar group) manually first to check it’a not foobarred after the update, then crack on with the rest. Testing a restore is 1 thing, but restoring the whole system…?
+1 for ansible.There’s a module for almost everything out there.
All my services run in podman containers managed by systemd (using quadlets). They usually point to the :latest tag and I’ve configured the units to pull on start when there is a new version in my repository. Since I’m using opensuse microos, my server (and thus all services) restart regularly.
For the units that are configured differently, I update the versions in their respective ansible playbooks and redeploy (though I guess I could optimize this a bit, I’ve only scratched the surface of ansible).
Personally I just wrote a bash script that does all of my regular updates and I run it manually whenever
And it’s stable enough for you? Do you go service by service or is it good enough for everything?
For docker compose I have a part of the script that gets all subdirs of “projects” dir and for each one does an update (that way any new service will be updated without having manually specify in the script) for everything else I just hard coded the update process.
Generally 90% of my updates are just running the script, on the other 10% I do some manual work (like updating configs, etc)
But for the most part this is me refusing to use already existing tools that could probably do most of this better
cd appname && dockup && cd ..Dockup being an alias for
docker compose up -dRepeat for the few services I have.
I don’t understand. docker compose up starts the container. When does the docker compose pull happen? Or is there an update directive in the compose file?
Whoops, I forgot that the alias includes a pull for the latest versions.
Ah. I thought there is an option in docker compose I could use.
And a docker image prune - a while the containers are running? :)
So everything is dockerized and points to :latest?
What about the necessary changes to the docker compose files? What about changes necessary in nginx configs?
I guess you also read each release notes manually?
One of the reasons I switched to YunoHost (the other being backups).
Proxmox community scripts has some nice update tools
I wonder if anyone ever wrote an update aggregator that would find all package managers, containers and git repos and whatnot and just do all of them.
Some are a right pain to update, such as Nextcloud. Installing a monthly update should not feel like an enterprise prod deployment.
It’s kinda ironic is that package managers have caused the exact problem that they are supposed to solve.
I am developing a script which will do that specifically for my services.
Right now at the first stage it only checks GitHub, Codeberg, etc. To check if there is a new version compared to what each service is running right now.
https://git.jeena.net/jeena/service-update-alerts
I am extending it now with a auto update part, but it’s difficult because sometimes I can’t just call a static script because some other migration things need to run. So I have a classifier which takes the release notes and let’s a local LLM to judge if it’s OK to run the automation or if I need to do it manually. But for that I am collecting old release notes as examples from each service. This takes forever to do so I only have it done for PieFed, PeerTube, Immich and open-webui, and I didn’t push those changes to the public repo yet.
I run most of my services in containers with Podman Quadlets. One of them is Forgejo on which I have repos for all my quadlet (systemd) files and use renovate to update the image tags. Renovate creates PRs and can also show you release notes for the image it wants you to update to.
I currently check the PRs manually as well as pulling the latest git commits on my server. But this could also be further automated to one’s liking.
