by Charlie Hall ’27
May 2024
The age of AI is developing before our eyes, and with it has come some websites that claim they can detect paragraphs with AI in them. Teachers and editors have been caught on the wrong side of the stick and now have to worry about students turning in work that isn’t theirs.
I decided to test the accuracy of these websites with an article written by a human, an AI, and an AI that I went through and slightly edited.
The topic I chose for my experiment was something I wrote in World Civilizations earlier in the year. I then gave the same prompt that our teacher, [Mackey] Luffman gave us to ChatGPT. I skimmed over the article that ChatGPT made, and to be honest, it was all right. I also took the AI article and rewrote some sentences that sounded weird to have a Human-AI mix, then ran those through several AI detectors to determine their accuracy.
At first, I expected that the detectors would not be accurate at all since, I didn’t understand how they could positively distinguish what was real and fake. But the vast majority of the detectors were correct.
The way these detectors work is fascinating. They are given a set of human and AI texts to train them; they use that information to see which one the essay I give them most resembles. SEO.AI describes how they train their detector as “rely[ing] on large datasets of human-written and AI-generated text to make accurate predictions,” which is almost exactly what I did to test their accuracy.
I expected that the AI essay with human edits would show up with mixed results, and in a way I was right. However, out of the seven detectors I used, only two confidently labeled it as AI (Quillbot and Undetectable AI). So even if a student were to generate an AI essay, they could spend 20 minutes rewording things and make it more human.
With many assignments like the World Civilizations paper I used, there is a long development process with notecards, sources, and criteria for this particular paper. Luffman isn’t evaluating just on the final product, but also on the process and the work that leads up to it.
It was evident that the essay the AI made for me was… not good, to put it nicely. You’d have to be desperate and foolish to use ChatGPT for 100% of an assignment. In theory, one thing that AI does well is unique vocabulary use. At the same time, I think that’s what gave it away that it was fake.
For example, here is a sentence from the AI generated article about baptisms:
“Originally conducted in the Jordan River, modern Christians often undergo baptism administered by a priest using water from a ceremonial vessel.”
This just seems like a really complicated way to describe a modern baptism. The AI is trying so hard to make something so simple sound as complicated as possible. This is part of the reason why people rather AI write their essays and companys like Grammarly use AI to improve word choice. Theres a fine line between having good word choice and sounding ouright rediculous.
I was especially scared that two out of the seven also labeled my real essay as AI-made (Quillbot and Scribbr). I’m pretty offended by this since Quillbot and Scribbr think I sound like a machine. If a teacher were to use one of these sites then there’s a chance a kid could get into a lot of trouble for writing a human essay.
It’s important to keep in mind that the websites were still right most of the time. If someone thought that a paper sounded fake, ran it through a few websites, and if several of them AI then it’s safe to say a student did not write that essay.
Some of the websites I looked at even had an option to put an AI essay in a generator and it would humanize it. I tried it and the results were pretty much the same.
Now, the AI essays were caught pretty quickly and easily. Only one out of the seven labeled it as human (ZeroGPT). I’m not sure what ZeroGPT was thinking. I felt there was something off about it. ZeroGPT did only get 33% right after all.
Two final takeaways:
First, don’t use AI to write your article — it’s not going to work and it’s dishonest and lazy.
And second, and most important, the bottom line is this: the human eye is the best AI detector there is.
