Had a backup script running via cron for months. Worked fine until it didn’t — turns out the disk filled up three weeks ago and the job started failing silently. Nobody noticed until we actually needed a restore.
The obvious answer is “check your logs” but let’s be honest, nobody’s reading cron logs daily for 15 different scheduled tasks across 4 servers.
What’s your setup for making sure crons are actually completing? Do you just grep logs periodically, or do you have something more structured? Curious how others handle this without turning it into a whole project.
Put the job in a script that only outputs text on error, and use MAILTO, like many have said. Due to spam filters that might not be an option though.
Next option is ntfy or gotify. make sure the script exits with nonzero status on error, and append
|| curl -XPOST -d 'cronjob failed' 'https://ntfy.sh/your-random-topic'to the crontab line. Naturally you’ll likely want the ntfy/gotify mobile app to receive the notodication. On iOS this might suck, as at least for me the ntfy app didn’t update in the background at all.Next one is an push uptime monitor, self-hosted or managed. I like Uptime Kuma. Make sure the script only exits with 0 status on success, and append
&& curl 'https://your-monitor-endpoint/identifierto the crontab line. Or use an inverted monitor and use||instead of&&.The uptime monitor will then notify you if the script hasn’t succeeded in the defined interval.
With some combination of these you can be pretty confident you’ll notice a failure, but still better to check every once in a while.
I also have a lot of jobs around that notify for each run, despite success or error, but it’s a bit too easy to just glance the message and accidentally ignore an error.