Should You Write Your Essays With ChatGPT?

There’s been a lot of hype around using ChatGPT and how it’s going to absolutely change the world, and a lot of people have been blown away by how well it can write. There’s a lot to get excited about, and a lot of controversy around whether or not students are suddenly going to be cheating on all of their homework assignments, and writing their essays using ChatGPT. No doubt you’re asking whether or not you can use it in your day-to-day life and save yourself a couple of hours!

It’s probably only good enough for a Grade 9 English Essay

Why?

When you take a look at content written by ChatGPT you can tell it follows a certain style unless prompted to write as if it were a specific author. Even when prompted to write in the style of specific author, for example Dr. Seuss, it will generally play things pretty safe, avoiding controversial topics or language. Once your reviewer or professor has seen one essay from ChatGPT, they’ve seen them all. If you pull a class of 30 students their writing styles will naturally be much more varied.

If we ask it to write a poem in the style of a more complicated author, like James Joyce, it doesn’t do so well.

It’s definitely not a bad poem by any means! But it’s certainly not in Joyce’s style, which is extremely creative and experimental. His style is just too far away from the standard writing style (and further still from 99% of written content on the web like magazines or news pieces).

The second issue with using ChatGPT is the content of the essays and this is especially true once you get into more technical content. It can very easily create a high level essay detailing the pros and cons of let’s say hydrogen as a clean fuel source, but it only pulls in surface-level knowledge. There is no mention of deeper issues with the technology, and definitely no consideration of second-order effects. We can see this in the sample below:

Some major problems in this essay right away - Pro #2 lists that we can produce hydrogen from renewable sources, but since this is a balanced view it would be worth noting that 98% of hydrogen produced comes from fossil fuel sources. If you’re trying to be persuasive of course you could omit that, but a balanced take would mention it. Pro #3 is factually incorrect - the energy per kilogram is really high, but the energy density (per unit volume) is quite bad. This is the main drawback to hydrogen, that you need massive volumes, or you have to compress it.

Third, we can see in a lot of content that ChatGPT just can’t pick a side! Normally in an essay you’re being tasked with defending a position, presenting the most persuasive arguments possible, or showing hard technical evidence. ChatGPT almost always throws in wishy-washy special considerations for its recommendations:

“While many people think X might be true, it is important to consider W,Y,Z and how these factors might impact the local community”

If you’re trying to persuade someone you generally don’t put in special considerations like that since it weakens your argument. You don’t want to come out and say “I think I’m right but the other team is also right”.

Why does it write this way?

 It’s a Generative Pre-trained Transformer (GPT) model, meaning it sucks in a lot of data from various sources on the internet (including some hand-labelled data from the creators), and then tries to use that data to come up with something new. You can see some of this technology in the AutoFill function on your phone, suggestions from Google, and the suggestions when you’re writing an essay in Word. The first word in a sentence could be anything, so it’s hard to predict. If your first letter was “T” however, there’s a high probability the next letter is “h”, then “e”, so we can reasonably predict you want the word “The”. As you fill in the sentence the probabilities of the next word narrow down, and prediction gets easier and easier. Suddenly we can apply statistics to language.

If you want a computer or an AI to generate an entire sentence for you, that becomes more difficult. Is it going to know where to place nouns vs verbs vs adverbs? If you give it enough data, the answer is yes. It just becomes a matter of statistics at that point. If all the data its trained on shows this word before that, or a certain type of sentence structure, it will recreate that. What comes out the other end is review by humans, and this leads to a more accurate imitation of language.

The major problem for you as a user of this technology is that ChatGPT is not going to create distinct essays for each person prompting it for one. It’s going to pull the same data, and write in more or less the same style, without creativity, and create more or less the same essay. I think you can get away with this at maybe a Grade 9 (~14 years of age) level, with a professor that’s not familiar with ChatGPT’s style, but above that it just doesn’t go deep enough.

There are some talks of enhanced models coming out later on that will use ~5x the amount of data to train, but until those are released I don’t think this tool is good enough to replace a competent human.