Previous posts: https://programming.dev/post/3974121 and https://programming.dev/post/3974080
Original survey link: https://forms.gle/7Bu3Tyi5fufmY8Vc8
Thanks for all the answers, here are the results for the survey in case you were wondering how you did!
Edit: People working in CS or a related field have a 9.59 avg score while the people that aren’t have a 9.61 avg.
People that have used AI image generators before got a 9.70 avg, while people that haven’t have a 9.39 avg score.
Edit 2: The data has changed slightly! Over 1,000 people have submitted results since posting this image, check the dataset to see live results. Be aware that many people saw the image and comments before submitting, so they’ve gotten spoiled on some results, which may be leading to a higher average recently: https://docs.google.com/spreadsheets/d/1MkuZG2MiGj-77PGkuCAM3Btb1_Lb4TFEx8tTZKiOoYI
This is what I was suspecting. People are just guessing. I have had people adamantly claim pictures that I took were AI generated and that I was lying when I said they weren’t. I see people on the internet constantly pull reasons out of their ass of why something is AI generated, saying “a real artist wouldn’t make this mistake,” but my wife has made those exact mistakes before.
I feel for you. A few people said the human art I put in the survey were lackluster but I thought they were pretty good, not everyone is an S-tier artist.
77% of people guessed this was AI generated, and a friend of mine kept saying it was weird and inconsistent so “I doubt a real artist would put random food in the back”
It’s actually a cropped image of https://www.deviantart.com/tsaoshin/art/Strawberry-Taiyaki-Cat-905271835 . I wouldn’t want to be an artist right now.
Anyone who says “a real artist wouldn’t …” Needs to spend some more time at the gallery.
Anyone who says “a real X wouldn’t…” where X is any profession, is really clueless about anything that X do. Though it is particularly true for artists, I’ll admit
I feel like most professionals in a discipline say ‘a real X would absolutely be so stupid as to…’ instead. They have seen every mistake the profession can offer.
Yes, developers would absolutely delete production databases even after you told them to triple check. Yes, artists absolutely would start a world war over racism. Yerp, archers would absolutely try to off themselves with a bow. And oh dear would car drivers absolutely drive into oncoming traffic and honk because everyone seems to drive on the wrong side.
People make mistakes and do stupid shit all the time. Maybe we should treat them like we treat AIs, just throw them away when they fucked up? 🙃
That image is adorable. It was one of my favorites from the survey.
I would love to see the same study but with every image being ai generated.
That or even every image being human generated. See all the reasons people come up with to explain why they thought human generated art was AI generated.
Would be good to see tests against only images of real life scenarios or things. Limited to photorealistic people, places, events like say floods etc. I think cartoons or non photorealistic drawings will always be hard to pick as there is not an obvious image of reference you would have in your mind as there is no limit to different artists styles.
Yeah. These examples were cherry picked to mislead. Pick a set of random images, and the result will be totally different.
I feel like without the usual bad hands tell, it’s pretty impossible to tell. There’s a few other “tells” but your image selection is particularly avoident. Having generated plenty of images, cherry picking the best is par for the course.
Although, a bad ai sample is pretty easy to tell at least. Even up against a bad human artist, a person with 8 fingers is easy ai giveaway. And for every one good image we get from SD at least, I toss out a few dozen.
I got the cartoon with text wrong because I didn’t know dalle-3 could do text. I got 12 from vibes of more detail? This assumption that AI does more detail gets a lot of actual humans. AI image generation is really crazy. The simpler stuff is pretty much indistinguishable from real human stuff at this point.
I can’t tell the difference at all between human-generated digital art and AI-generated.
Can people tell tell
Right down the middle.
Is like to see more of these tests. Maybe mix the formula up a bit
I just tried the survey and got 12/20, it’s interesting looking for the details that give it away and how often that lead me to the wrong answer. Comments on each image below because I thought this was neat, hopefully I’m using spoiler tags correctly.
spoiler
Image 1 (Incorrect): I was totally 50/50 on this one, and I guessed wrong.
Image 2 (Correct): Glasses look wrong when looking closely. If he wasn’t wearing glasses, I think I would’ve been 50/50 on this one too.
Image 3 (Correct): The most obvious AI generated one in the list.
Image 4 (Incorrect): I accidentally spoiled myself on this one by clicking on your image before the survey, but there’s no way I would’ve guessed this was AI so I chose “No” to keep my result accurate.
Image 5 (Incorrect): I thought the shoes looked kind of odd, that was the only thing I was going off of.
Image 6 (Incorrect): I was really conflicted on this one. The floor of the boat didn’t look quite right, but not necessarily in an AI way. In the end, I thought it was AI because of the random pieces of wood in the boat. That seemed like something an AI would randomly add, but I was totally wrong.
Image 7 (Correct): Eyes are clearly AI generated, and parts of the sword look off. If it wasn’t for those, it would be really hard to tell.
Image 8 (Correct): The butterfly has an extra wing coming out of its back.
Image 9 (Incorrect): This was another one I was conflicted on. I chose AI because the dirt(?) right after the waterfall seemed odd, but I was wrong.
Image 10 (Incorrect): I was fairly confident in my guess. The resolution makes it hard to tell, but it looks like right horse is melting in that usual AI way, but I think that’s actually supposed to be the thing that connects the horses to the cart at a second glance.
Image 11 (Correct): I couldn’t imagine an AI drawing trees like that.
Image 12 (Correct): The break in the river in the middle of the image looks odd.
Image 13 (Incorrect): I was 50/50 on this one, guessed wrong.
Image 14 (Correct): This is the other one I spoiled myself on, but the highlights look super weird. I’m quite sure I would’ve guessed this was AI-generated regardless.
Image 15 (Correct): I didn’t think an AI would generate an image with specific details like the fish bowl or Pocky that well.
Image 16 (Correct): I almost got myself thinking this was a trick question because there’s no way an AI would generate something that accurate.
Image 17 (Correct): This generally looks AI generated, but the pattern on the pants was a dead giveaway.
Image 18 (Correct): The arm rest is at the same level as the chair/couch cushions?
Image 19 (Correct): I don’t really have any comments on this one, it just didn’t look AI generated.
Image 20 (Incorrect): This one has that AI-generated vibe, but I thought that might be intentional to make this harder. I couldn’t see any other details that made it obviously AI, so I guessed it was real. Welp.
This was a fun survey, thanks for putting this together.
Awesome job! Here is the first image dalle-3 generated of you.
Not overly convincing, but look at those hands!
Absolutely horrifying, thanks
Yeah why the fuck are the test tubes of different lengths???
The only odd thing I noticed is the test tubes that look like they should be out of focus like the rest of the background. That is extremely convincing and idk if I would notice if I saw it out of context.
My wife and I took part in the survey as people who aren’t in programming and have never used LLM art generators before, and we scored quite poorly. I got all of the photorealistic images correct, but the painting/drawing pictures were much more difficult.
I think a better test would be to give four real artists and five LLM art generators the same prompt, show all nine of those together in a square and you have to pick the real ones. Then we’re comparing like-for-like as opposed to trying to spot an LLM image out of the blue.
For the sake of an survey you are right.
But practically this shows how easy people are fooled nowadays, and it will only get worse in the future.
I scored 9 on this so I’m right at average. lol
I was quite disappointed that I “only” got 14, originally, but these results make me feel a lot better about that score. Any chance you could share some more involved results of this survey? I’m curious about percentiles for each score, and standard deviations for those averages - is 9.70 vs 9.39 statistically relevant or within expected variance?
I was, of course, fooled by the cartoon and the lineart sketch, those two are stupendously convincing.
Someone asked for the raw dataset in the other thread, here it is: https://docs.google.com/spreadsheets/d/1MkuZG2MiGj-77PGkuCAM3Btb1_Lb4TFEx8tTZKiOoYI
Have at it.
Thank you! I’ll poke around the data.
