Had a backup script running via cron for months. Worked fine until it didn’t — turns out the disk filled up three weeks ago and the job started failing silently. Nobody noticed until we actually needed a restore.
The obvious answer is “check your logs” but let’s be honest, nobody’s reading cron logs daily for 15 different scheduled tasks across 4 servers.
What’s your setup for making sure crons are actually completing? Do you just grep logs periodically, or do you have something more structured? Curious how others handle this without turning it into a whole project.
Plus one dir a simple alert wrapper. Although for me it’s even more simple: the only cron stuff I have is backup and individual service updates and both have an entry on my Homepage Dashboard with a health indicator. Red dot == shit.
The only fancy custom thing I’ve built is not even cron based: sometimes I missed on my nixos server that I needed a reboot because kernel modules updated as well - so my nushell prompt compares the live environment with the loaded kernel - if versions mismatch my shell (rightfully) flames me to pay more attention and that I need to reboot.
For anything from based I’d use a wrapper - for me it would be a simple curl to the localhost matrix bridge so that I’d get a push message but for others it’s Mail or just a ntfy - whatever is easiest for your server to reach you, really. I would not want to check on my automations and the notification service id notice within two hours if it’s down because I use it for a lot of other items as well (no mails, appointments, reminders or events in 2 hours??)