So my manager today asked me if I could stay later when there’s broken things in prod, and then today his star dream employee yolo’ed a full stack change into prod without review. It’s fucking massive and implements new API endpoints, touches >20 files. Many of the diffs are too large to render in the browser.
It’s almost comical, but something immediately broke.
Most of my day, I’m digging through code to identify bugs created from this shit, just to get a stealth merge midday.
I kind of don’t know what to do.
I say as long as Ai exist, real programmers will always have a job and are not replaceable. YOLO Vibelords are replaceable, but only real programmers have the skill to go through the shit. That’s the most secure job ever if you ask me.
I kind of don’t know what to do.
Rollback and go home
I had a colleague who pushed a load of half cocked broken nonsense to a repo i was owner of. Blatant emoji comments from AI and all. I just reverted it because I was under a load of pressure to get it out and on ASAP.
He messaged me 6 months later asking me if I “deleted his code”. I told him I reverted it because he pushed a load of broken bullshit to main and swanned off for 6 months.
He was like, “oh…” then i realised. This guy doesn’t even know what git revert is. Had to explain it.
What the actual fuck.
Don’t provide bailouts for neither your manager nor your colleague. Highlight as far up the chain as you can the damage their behavior is having on the business.
Roll it back and go home.
Do NOT under any circumstances fix it. The other guy needs to and you need to be clear he broke it and how
I would immediately fire somebody with such a mentality. Letting bugs live in production just to prove your point, that’s borderline criminal. I get you’re not happy with the vibecoder’s job but there are other, smarter ways to deal with this than sabotaging the company.
Gunna be honest, if you really feel that way you’d fire the vibe coder. I don’t give a shit if it’s broken ifs due to the negligence of management and other employees.
Poor planning on your part does not constitute an emergency on mine. And ofc this varies based on the service as all things do. But it’s rarely important enough that I’d stay over. Like if lives where at risk or it’s a “bring the company down but” maybe I’d stay. But there would be hell to pay for whoever broke it later. Either that or the company would loose me as an employee
These bugs shouldn’t have reached production. This is why you do reviews in chunks.
Probably very true (again, it depends on the product/company).
Other ways such as?
I dunno, human things like communication and letting people know what’s wrong here and how not to let that repeat. If it’s pathological then maybe that work environment is not fit for you.
By that logic wouldn’t you be firing the vibe coder for not rolling back the commit?
Maybe. Context is not sufficient (mistakes can happen, intention counts) but breaking prod may be bad depending on the product type.
Fallback and fix is the best solution.
I came here to say exactly this. The offending dev and your manager aren’t going to understand the severity of issue until he spends days trying to find the bugs. Not your circus, not your monkeys.
You need SRE concepts. First, if you break it then you fix it; in a system where anybody can make a change, it’s the changer’s responsibility to meet service objectives. Second, if your boss doesn’t find that acceptable then they need to appoint a service owner and ensure that only the owner can make changes; if the owner breaks it then the owner fixes it. Third, no more than half of your time should ever be spent fixing things; if something is constantly broken then call a Code Yellow or Code Red, tell your service users that you cannot meet your service levels, and stop working on new features until the service is stable again.
Under no circumstances, ever, should anybody stay late. There should only be normal business hours, which are best-effort, and an on-call rotation which is planned two months in advance. Also, everybody on call should be paid hourly minimum wage on top of salary for their time.
Ask to be paid by the hour
You render diffs in the browser?
And no, let the yolo fix the things he broke.
browsers are quite capable of rendering text, even images!
But they’re bad at it. Especially in performance and amount.
Not true. Browsers are definitely faster handling text than your average terminal emulator without GPU capabilities, they just use more resources.
That said, GitHub diff viewer specifically is a sluggish mess. But that’s not because of the browser.
What, you think the usual text editor renders all lines just because? No, they load the lines that are currently visible + some more for scrolling. At least the ones that don’t choke on a mere 100k lines. And they don’t need GPU for this.
But webbrowser engines just aren’t made for this, but for DOM rendering.
So does a browser. Modern browsers don’t even need to receive the whole HTML to start rendering things.
There’s a lot of engineering effort put into browsers, much more than in terminal emulators because they need to do much more than just rendering text.
GitHub and similar tools show PR diffs in the browser.
Edit: I should explain first that I do not think you can change your employees or manager. A technical solution will never fix a management problem. Perhaps your manager is getting enough heat to be open to better management controls. You should be in “sanity protecting” mode.
Original: While the posts include a lot of fiction, but “overemployed” crowd have some good advice. Start applying elsewhere, downshift your effort at $job_one, and move to collecting a check.
Or ramp up and take your manager’s job or get fired trying. Gather data and allies in the C-suit, and stage a coup. Or unionize (or post pro-union flies in the bathroom).
Pushing to prod without review and breaking the running application is a resume-generating event in many companies. In many others it’s not even possible because of programmatically enforced policies.
If your company’s response is not to prevent or dissuade it but to have other people work overtime to fix the mess then that’s a major management fail.
Try to educate your boss about best practices. This incident should give your arguments some more weight.
Deployment to prod should not be something a developer can do by themselves; a proper CI/CD system can be configured so that prod can only be deployed to by people with an appropriate role (product owners or lead devs if your company doesn’t have POs).
If you don’t have such a system, make it an explicit policy: Only Steve the lead dev (or someone specifically appointed by him while he’s absent) can push to prod; if anyone else does it they get invited to an uncomfortable meeting with Steve. If they do it again the meeting will be with HR.
But seriously, you should lobby for a proper CI/CD system (if none is present) and for the system to be configured so that a) you can’t merge to the main branch without a code review and b) deploying to prod only works from main and with explicit approval by a PO/lead dev. That should stop most of the shenanigans.
This is not how you do development. Like, at all. If changing jobs is an option for you do it. If not, well, not much you can do. You’re not going to fix the entire company if management and other devs don’t know how it should work and don’t want to change.
The only solution: unalive them.
Make ci/cd part of the process. It stops a lot of people who push huge unmaintanable changes.
If it doesent pass the tests and cant build then it stops them from merging.
Yeah this is my general tool for dealing with shit code. I’m currently working on getting that into the CI
It helps! Ive seen it with two jobs completely change the culture around code review.
- Instead of you/team lead being the bad guy, its now the code/process.
- Code becomes more stable and releases actually become more frequent.
- If something goes wrong, your VM/docker/box/etc… can just be re-spun up by the same process.
- Easier to onboard since the same build process is in the CI. And is constantly being used rather than relying on the README (that may not have been updated in a while).
Mind you its not perfect of course. You can still “Vibe” test and/or remove tests that dont work. And make the project more brittle. Or go overboard with lint rules (I actually had to break up a fight with that one). But its much better than the old process of merge and pray.
When Rome is burning, do as the Romans like Nero do. Fiddle a nice tune, tell Claude to fix it for you, and go home. If they’re not willing to give any fucks, why should you?
Fix AI coding with AI coding. Genius. Also update your resume and start looking elsewhere.
If your manager won’t hear anything negative about Vibelord, polish up your resume. There’s nothing you can do to change the culture of your team if the boss isn’t on board with it.
Not sure what kind of person your boss is, but if you want to try to win him over, try to figure out what might sway him. If he’s technically illiterate, talk to him about best practices (like, I dunno, code reviews) and cite reliable sources and data if you think that would help. Avoid calling out Lord Vibington, the main thing is to put a picture into your boss’s head of what this could/should look like. He might not realize that these issues are preventable.
The goal here is to get your boss to take more ownership of the team’s culture, and start insisting on preventative measures. Mr Viberator will either have to conform, or there will be increasing friction between him and the boss.
