The makers of ChatGPT, OpenAI, have released a classifier to help people determine whether text was created by AI or a human. The tool could help mitigate false claims that AI was used or detect when AI is impersonating a human.
Unfortunately, the tool is not very reliable but in our tests it does compare better to other classifiers available. OpenAI wrote, “In our evaluations on a ‘challenge set’ of English texts, our classifier correctly identifies 26% of AI-written text (true positives) as likely AI-written, while incorrectly labeling human-written text as AI-written 9% of the time (false positives).”
We asked ChatGPT to rewrite OpenAI’s blog post announcing the tool, in a more formal style and then ran it through the classifier. The AI text classifier returned, ‘The classifier considers the text to be unclear if it is AI-generated.’ We then ran the same text through GPTZero, another text classifier which told us, ‘Your text is likely to be written entirely by a human.’
OpenAI are hoping that the tool will get better with time. They say, “We’re making this classifier publicly available to get feedback on whether imperfect tools like this one are useful. Our work on the detection of AI-generated text will continue, and we hope to share improved methods in the future.”
OpenAI Will Improve the Classifier Over Time
If the tools improve they will be able to help educators determine whether students are using AI to write coursework, detect and block political misinformation campaigns that use AI generated content, and possible labour disputes over whether AI was used to create content.
Andreas Theodorou, Software Editor for TechReport said, “AI tools like ChatGPT are exactly that… tools. The additional opportunities they offer all come down to the ways in which we use them, and the likelihood is that, as this technology grows and improves, so too will our use of it, until they are as much an extension of our working resources as spell-checkers,” he went on to say, “the real issue lies in the misuse of such tools. Those who would use it without due diligence and without the human input will only see it bite them in the long-run.”
While the classifiers currently don’t have a high success rate at identifying AI created content, the content that AI creates is still far from being usable in many practical scenarios. Even when AI creates usable content after the text is edited it can become impossible to distinguish between AI generated text and human written text. At which point, does it really matter who or what created the text?
