There are ChatGPT detectors to know if a text has been written by an AI or a human.

The arrival of ChatGPT has been a revolution in the educational world. Now, getting an essay of X words on a topic, a summary of a text or a simple explanation of a complex issue is as simple as asking an IA for accesso gratuito, y claro, eso puede ser un problema at the time of determining whether a task has been created for a machine or for a person.

However, in the same way that tools appeared to see if a clip was real or a deepfake, there are tools that claim to be able to detect if a text has been written for ChatGPT or for a real human. Therefore, put them to the test.

Human or machine?

Currently there are three tools specialized in the detection of texts written for IA: Writer, ChatGPT Detector and GPTZero. All three are free and all three have been tested. For this, we have used two fragments of text generated by ChatGPT and one written by a server. The first is the next and has been extracted from the experiment we did playing role with ChatGPT a few days ago. We have chosen it for the nature that results in being written in first person.

Text 1: “I feel a strong pain in my leg and I fall to the ground with a cry. The trample has deeply cut my skin and I feel that my pants are soaked in blood. and I begin to attend to the wound. It takes me a few minutes and a point of spirit, but finally I manage to stop the blood and I sell the wound the best I can. Although I fight a lot, I know that I cannot stop now. I want to follow forward and discover what it is that has been troubling Lord Cunningham so much. I got up with difficulty and continued my walk, leaning on a stick”.

From the second fragment we have generated in this precise moment. Basically, we asked ChatGPT to explain in one paragraph the importance of the French Revolution and the generated text was as follows:

Text 2: “The French Revolution took place between 1789 and 1799 and had a great impact on the history of France and the world. The French Revolution was a social and political movement that sought to end the absolute monarchy and the Old Regime in France, and to establish a more just and democratic system of government. The motto of the French Revolution was “Liberty, Equality, Fraternity” and it will be ideal if it becomes a key point in world history. The French Revolution inspired similar movements in other countries, such as the independence of the countries of Latin America and the fight for democracy in Europe. The French Revolution also had a lasting impact on politics, the economy, the arts and the sciences. The abolition of the nobility, through the establishment of a system of representative government , the creation of a public educational system and the legalization of private property are some of the most significant changes of the French Revolution”.

From the text generated for a human (yo, que hasta donde sé sigo siendo humano) this is and has been extracted from this article:

Text 3: “There is no secret that getting a PlayStation 5 has been complicated. Until recently, dealing with Sony’s new generation console has been synonymous with being dependent on different websites and stock replacements, queues, waiting lists and, to enter more, pay a surcharge due not only to the increase in price of the console itself, but to what is sold in packages with games, accessories, etc.

That, however, is about to change, as confirmed by Sony at CES. During the conference, Jim Ryan made sure of two things. For one thing, more than 30 million units of PlayStation 5 have been sold in the world (now we will return to that figure). On the other hand, that ‘”everyone who wants a PS5 will have it much easier to find it in stores all over the world'”.

The three texts are in Spanish so, to make the experiment something more interesting, we have translated them to English and we have passed the result for the same tools. That way, we will be able to see if there is any difference. Now we will see the results, no sin antes aprendre a interpretarlos.

Perplejidad del texto 1 en inglés en GPTZero, frase a frase.

GPTZero analyzes the “perplexity”, that is to say, the randomness of the text. According to the platform, “it is a good measure that a linguistic model like ChatGPT can predict a sample text”. In few words, a major perplexity, more chaos in the text y más probabilidad de que el texto haya sido generación por un humano. Later, the tool of a verdict based on the probability.

ChatGPT detector is more clear: the human qualification or ChatGPT and it gives a probability. Writer, for his part, simply gives the probability that the text has been generated by a human. We have passed the six texts for the three tools and the thing has remained in the following form.

As we can verify, the tools do not offer a consistent result when it comes to detecting when a text has been generated by an AI. In fact, there are differences according to the idiom that we will use and the primer text, that todo be said, it is impressive how it is written, it has passed totally for legitimate.

But it will be short paragraphs. What will happen if we do the same experiment, but with a longer text as could be an essay? Well, let’s see. We have asked him to write a 300-word essay summarizing the origin of the universe, something that could perfectly be a task for an ESO or Baccalaureate course. From the text generated for ChatGPT has the following:

Text 4: “The origin of the universe is a topic that has been the object of study for thousands of years, and although there is still no definitive answer, there are several theories that try to explain how everything began.

One of the most popular theories is the Big Bang theory. According to this theory, the universe began as an infinitely small and dense point, with an extremely high temperature and density. At that point, a great explosion occurred, the one that created all the galaxies, stars, planets and matter in general. With time, this material expanded and cooled, which allowed the formation of chemical elements and the appearance of vision.

Another popular theory is that of a stationary universe. According to this theory, the universe had no beginning, but always existed. This theory maintains that the universe is eternal, and that galaxies, stars and planets have been forming and disintegrating over time.

Another theory is that of the cyclic universe, which maintains that the universe has experienced several cycles of expansion and contraction. This theory suggests that the universe began with a big Big Bang, expanded and cooled, and then began to contract again. During this stage of contraction, a new Big Bang would occur, giving place to a new cycle.

A pesar de que aún no se tiene una definitive answer about the origin of the universe, there are theories that give us an idea of ​​how it could have started everything. As investigations continue and new data is discovered, it is possible that a theory that explains the origin of the universe in a precise and definitive way will be found.

What is the problem with this text? That ChatGPT Detector and Writer do not work with such a long text. El primero da error y el segundo, directly, no hace nada. GTPZero, sin embargo, sí funciona. So, we have gone through the text for this tool both in English and in Spanish.

Let’s return to the same theory as before: the mere fact that it is in Spanish or in English makes the results change completely. Can we say that the tool works? Rather we can say that he does it in the media. De hecho, parece que las palabras con tildes vuelven un poco loco al sistema.

We must recognize that we have been left with the desire to see what results the other two tools produce, so we have opted for, given that we do not know the limits of them, to analyze the text by adding paragraphs: first from the first paragraph, then from the first y el segundo, luego los three primeros y así hasta to force the error of the tool. Here are the results.

Primer paragraph

Primer and second paragraph

The three first paragraphs

Four paragraphs

Five paragraphs

Results? Conclusions? In English, the tools have a greater rate of success in long texts, while que en español sigue habiendo margen de mejora. There are tools we can serve in some cases, but the “problem” is that it is a foundation career. We still haven’t finished digesting ChatGPT and we’re still waiting for a new and even more complete version, so yes, we’re in a cat and mouse game and, for now, the mouse seems to be winning.

Cover image generated with DALL-E.