I have a tool that I wrote, probably 5+ years ago. Runs once a week, collects data from a public API, translates it into files usable by the asterisk phone server.
I totally forgot about it. Checked. Yep, up to date files created, all seem in the right format.
Sometimes things just keep working.
Yeah, all these simple data processing scripts will always work as long as both sides stay the same/compatible
Yep. It seems they haven’t changed a thing about the format. Probably a script much older than mine on their end is generating it too.
Isn’t that true for all of data processing?
Maybe. But webdevs have made it a mission not to seem like so
Meanwhile, had to debug a script that zipped a zip recursively, with the new data appended. The server had barely enough storage left, as the zip took almost 200GB (the data is only 3GB). I looked at the logs, last successful run: 2019
Need some monitoring!
Oh no need. The client didn’t noticed anything in 6 years, and the reason why we had to check is because they wanted us to see if we could add this feature… That already existed.
My favorite part is, if you do some extensive analytics from time to time (e.g. to prepare an upgrade to a new major version) and as a side effect stumble upon some workflows/pipelines/scripts constantly failing (and alerting the process owner) every five minutes for… at least a few months already.
Then you go and ask the process owner and they’re just like “yeah, we were annoyed by the constant error notification mails, so we mad a filter that auto deletes them”…
Yes, had the same happen. Something that should be simple failing for stupid reasons.
Well it’s not that simple… Because whoever wrote that made it way too complicated (and the production version has been tweaked without updating the dev too)
A clean rewrite with some guard clauses helped remove the haduken ifs and actually zipping the file outside of the zipped directory helped a lot
I mean, I have to say I’ve hastened my own demise (in program terms) by over-engineering something that should be simple. Sometimes adding protective guardrails actually causes errors when something changes.
Am I understanding that last part correctly?
[…] and actually zipping the file outside of the zipped directory helped a lot
Did they just automatically create a backup zip-bomb in their script‽
I oversimplified it but the actual process was to zip files to send to an FTP server
The cron zipped the files to send in the same directory as the zipped files, then sent the zip, then deleted the zip
Looks fine, right? But what if the FTP server is slow and uploading take more time than the hourly cron dispatch? You now have a second script that zip all the folder, with the previous zip file, which will slow down the upload, etc…
I believe may have been started by an FTP upload erroring out and forcing an early return without having a cleanup, and progressively got worse
… I suppose this happened. The logs were actually broken and didn’t actually add the
messagepart of the error object, and only logging the memory address to it
the final part of that is “written by person that left the company ten years ago”
A self-written shell script “daemon” that tails & greps log output for “ERR|FAIL”
Nah bro, that bash alias is FULLY documented in .bashrc! Idiot.
I don’t see the alias in your .bashrc
yeah, um, about that. I have no idea where it comes from. We can type alias and see what it is, so if it’s ever lost, we can recreate it, but I looked for 30 minutes yesterday even did a grep -R and I have NO IDEA where it comes from, or why it’s named electricboogaloo
My current project has a crontab with 216 entries.
Well, here’s a sentence I haven’t been tempted to use before:
“I believe that may be too many crontab entries.”
just randomly delete 50 of them.
Yes. The strongest crontab entries will probably restore themselves. (For anyone reading along, this is sarcasm. Don’t do this.)
a crontab can regenerate from bisection to form two whole crontabs
Any problem in server administration can be solved with an additional crontab entry. Except for the problem of too many crontab entries.
And that’s why I added a crontab entry that periodically purges my cron configuration. That way, I’m forced to readd only the truly necessary cron jobs, successfully reducing the amount of crontab entries.
OK, I got called out
Suck my dick O’Leary
