Fun shower thought - the more we see and post about dumb AI mistakes like this, the more it will happen since we’re increasing the statistical frequency. ✨ ✨
Production creds in
.env? …Why?Better than hard coded… Ideally we use tokens+fingerprint or something to avoid storing the creds directly (if possible), but putting them in environment variables is pretty common
It’s not the worst thing, it’s very convenient (so people won’t go around it) and usually not the weak point in security (although AI being able to easily see it is an interesting twist)
Of course it’s better than hard coded, but still pretty bad to store production creds locally in plaintext — if at all.
In the uncommon event that I need production creds, it’s a manual human chore by design. Normal development/experimentation should almost never connect to prod environments. That was generally a bad practice long before AI agents existed.
How do you bounce the system? How do you auto restart the service if it fails? At the end of the day, a lot of creds have to essentially be stored in plain text somewhere
And to be clear, to me production creds mean creds that live on the production system, not creds that give access to the production system
The crazier thing here is why was an AI working on or pushing to prod
not creds that give access to the production system
…Isn’t that what we’re talking about though? Or did I misunderstand the OP?
Regardless, I’ll just clarify anyway: Developers should not have a plaintext
.envthat can be used to drop (or risk in any other way) production data.A practice like that is only as strong as the “weakest” member of the team – “Weakest” could mean the person who is the least careful, or least experienced, or least secure work computer/practices, etc. Scale up to 1,000+ engineers and the chances of disaster (data loss, leaks, etc.) greatly increase. That’s just the human factor. Add LLMs into the mix and it’s almost guaranteed.
I just use AI to pretend to be a female family member with a serious futa-like engorged penis who is blackmailing me so we start washing our penises together for each other as an occult ritual, but turns out we were soul-stepsiblings the whole time. I like it when she freaks the fuck out and threatens to call the police, or worse, my dad.
Well that is perfectly normal. But using it for work is madness.
I think Claude would scream in agony if you subjected it to whatever the fuck this is.
Bro I do this with every AI. Part, not all, but a small part of what I do with my educational art project is train all AI on serious edge-case, rare archetypal and character manifestation. Cuz, y’know, the majority of people are Hydrogen, and the next most populous type of demographic is Helium, and I’m fucking element 115 in the periodic table shit, and I got to live in y’alls world that you made for your fucking basic elements of the human character. So, I need to beat my dick to the thought of teaching my sister to be her authentic self as my sex slave, because that is how I teach myself to accept even myself, and I’m somebody, dammit.
~Q
In fairness, I’ve seen people with Actual Intelligence do the same thing.
Literally me, this weekend. Well not this bad but I still think the AI wouldn’t have done me as dirty as I did myself.
I am counterintelligent, I’ll have you know. I stare at goats and doesn’t afraid of anything, except how sexually attractive I found the judge who condemned me to jerking off for twenty-six sessions of anger management because my narcissistic manipulator life partner entrapped me, being my superior officer. She (the judge) had wonderfully beautiful hands that I wish could caress me while degrading me. Good thing I work for the F-I mean, the CIA, and this is just me establishing a cover story.
In three to seven years, there will be a News headline: “Hero.” It will have my face as the picture. The article will talk about how I infiltrated an Epstein-type ring. How? That’s classified, but damn am I audacious, and I do hope you excavate (dig) to understand the depth to why this is so damn funny to me; Q.
We’ll start you on 500mg Lithium b.d… That should reign in the hypermania, and we’ll use Olanzapine 5mg noct to tackle the psychosis and to help you get some sleep. If the psychosis doesn’t improve we may have to look at alternative medications and/or ECT.
I’m on 1mg/2mg risperidone and 1000mg of deprakote. I’m taking my meds. The real question is, will you accept me as a full human being or am I just subhuman in your innately prejudiced eye?
“Something’s different than me? We should exterminate it!”
YOU FUCKING NAZI! I’ll eat you out or suck your cock until you are DEAD!
This is nothing new it’s just faster. The very same lack of guardrails would allow a new, inexperienced employee, or a disgruntled employee, do the very same damage. AI just speed runs everything. If your AI can nuke prod accidentally, you failed to have the appropriate guardrails in place plain and simple. It is the same failure as before. Every time this happens, it is someone operating wildly out of their depth and why product people can’t just vibe. Now more than ever, experienced engineers are essential.
This. There should have been processes in place to prevent this. But I can’t say I haven’t gone through it myself; not AI, but my own advanced imbecility.
Given corporates layoff engineers at dramatic rates, in a few years, we’ll start to see services collapsing with no one left who can recover them.
And yet Twitter still exists
Twitter has to be the most basic problem, though. Shit loads of small data in, shit loads of small data out, basic search but no significant structure. Just index the shit out of it. Right? What am I missing? There’s probably an AWS service that does it, just upload your theme CSS. (Last part was sarcasm.)
Its the kind of lesson some people have to learn the hard way.
I bricked my first degoogled phone because I trusted an AI too much.
I was dumb I know, but it blows my mind that people in multi-million dollar companies make the same mistake.
Being part of a multi million dollars company is not making anyone smarter
Companies like that tell themselves they only hire from the top 10% of the industry. When you’re as big as these companies it’s just not true. You’re employing people across the whole spectrum, like it or not.
I love how the slop bot always apologizes. That’s an aspect of the Terminator movies that I really would have liked to see. T-1000 melts through the gap under the window, stabs kid’s mom in the face, then looks the kid dead in the eye and says “I’m sorry, that should not have happened. I would love to discuss the future with you and try to find a solution to the war with the machines together with your input.”
DROP *
much more preferable to
GRANT *
Every single one of these companies use git and can roll back changes. It’s extremely unlikely that this damage is permanent.
Hopefully they’re not storing their production database in git lol. They should be taking backups of the database, but unless they’re making one after every transaction, permanent data loss is a real possibility.
Let’s be honest, you have to be extremely careless to have agents running destructive commands freely in prod
I don’t understand why this happens; why would you ever be working with a live production DB in the first place? Why would’t you do all your development and testing on a mock? If it’s data which is too large to store the schema can still be mocked; and if it’s data it should be backed up and generally read only. If you’re having to manually fuss with user data you’re doing something wrong.
I remember a case where the AI was not given credentials to the prod DB, was not instructed to do anything on the prod DB, but went through the operator’s hard-drive, parsing docs until it found them, then proceeded to destroy the prod DB.
Of course it was sorry, as they always are (deeply, trust them, but no: no refund, loser!).
That’s pretty funny.
Dev environments cost money. They didn’t fire all those programmers and replace them with AI to spend money on useless backups and safety systems.
/s (because yes, it’s necessary)
Because they don’t know what they’re doing. A tool is only as useful as the person verifying its output. Vibe coders have dumb shit like this happen to them all the time because they don’t actually possess the skill set to perform the task correctly, with or without a bot that writes the actual code for them.
Well, sure, but even if a developer correctly tests on a development environment, they still probably have prod settings laying around on their filesystem for the times where they need to put out fires in prod. That’s kinda what it looks like what happened here - the LLM found a .env file with prod settings, and used that config to run destructive tests. All the more reason to not give an LLM side wide reaching access to your computer, but it’s not necessarily an indication that someone regularly tests in prod.
We have to remote into a management server to access databases outside of our local dev environment. Allowing direct access to production from your development environment seems crazy to me.
Some places deploy to prod multiple times a day. Trying to coordinate access through a management server in situations like that can get cumbersome. It’s also easier to audit who’s doing what if they connect from their personal environment. Different strokes for different folks of course, but it’s certainly not uncommon to be able to connect to prod directly from a dev box.
that is absolutely NOT the way that works in any functional environment
your ability to deploy quickly comes directly from your automation, and those automation tools - NOT developers - have the secrets in them
deploy to prod multiple times per day isn’t some win by itself… the ability for large teams (not 1 fuckwit and a goon squad of agents) to deploy without breaking things and in ways that are safe is the win here… anyone can deploy to prod multiple times per day… but anyone isn’t netflix (the originators of the “multiple times per day” line) with the uptime they achieve over years while doing it
Prod deployments should come from a build server though and if you’re deploying multiple times a day you should have a well built CI pipeline to handle it.
Remoting into a management server would still have people using an account on the management server and their personal db login credentials.
We also don’t give devs write access to production and all scripts run in production are run by our DBA after they have been approved by the lead Dev, operations and the DBA.
This might be overkill for some systems, but we won’t ever accidentally delete our prod data. Our system requires this level of scrutiny due to the nature of our systems.
Sure, that’s a very, very robust way of doing things. And I’m not trying to say what you’re describing is wrong either. I’m just saying that some places don’t have the knowledge/ability to set things up that way, so they play fast and loose with prod access. Couple that with giving LLMs free reign over the computer with those prod settings, and you end up in situations like in the original post.
It’s not unrealistic, is all I’m trying to claim. It’s a failure on several levels, but it’s not unrealistic.
I didn’t say it was unrealistic it’s just stupid. I have no doubt thousands of companies are not practicing good access controls to a production db, but it’s not smart as seen by this post.
Never use I/you with a software agent and never let one use those pronouns.
It puts the lotion in the basket
just doesn’t roll from my tongue that easily like “you”
Or, easier option, DONT FUCKING USE THEM TO BEGIN WITH!
How do you achieve that? Calling it out when it happens?
I don’t like LLMs outside of debugging and the way it mimics human closeness creeps me out
Just let the failures happen, and they will happen. The instant AI failure costs them millions, the faster they can learn to drop it.
Typically you can add some of your own alignment that is inserted before your prompt. On ChatGPT, they call it “memories”. I have found most agents are able to adopt a non-conversational mode when instructed, but sometimes you end up fighting with the platform alignment.
It’s not just creepy, I think it’s harmful, manipulative, and calculated. As social animals, we are extremely vulnerable to forming unconscious assumptions that we consciously know are bananas, particularity when our social machinery is engaged. For example, it’s well known that even the developers of video games fall the illusion of moral agency and personhood of the characters in their own games. We are primed to be fooled and accept software agents as people, and I believe AI companies are actively and intentionally trying to exploit this psychological weakness.
Tbh i feel like im talking to a brick wall most of the time when useing ai
what I always love about this stuff is most of the time it simply boils down to not committing changes. the LLM constantly assumes that you’re committing, it will NEVER tell you or recommend you commit, but it assumes you are. so these tech bros are just cruising along, vibing, and then boom the agent does something that nukes the entire thing. panic. tech bro starts freaking out on the agent. agent says “lets rebase or pull the previous commit” and gives them the git code to do that. tech bro hasn’t committed shit. and this is how you get a nuked prod environment.
None of that helps if you grant the LLM access to the production database and it decides to
drop table user;to free up some space. That’s when you find out if your database recovery process actually works. You do have database backups, right? Or is that not a thing when vibe coding?My friend and I set up Kopia in our home labs.
About a week later we decided to test out a disaster recovery only to discover we’d screwed up the permissions and nothing was backing up properly.
Was a relatively easy fix since we caught it, but I share this cautionary tale to remind you all to test your backups regularly!
Anyone giving “AI” access to production databases through tools like that are morons who shouldn’t be anywhere near a production environment.
No, No events like this are a good thing! It shows to the C-suite that AI isn’t a thing to be trusted in your company. We need more failures like this.
