With the advent of chatGPT and AI text-generation software, the need for university tutors to design assignments which dissuade students from utilising these tools has never been more apparent. We can achieve this in part by setting second-order assignments which entail the application of a framework (or theory) to a specific text.

What is a second-order assignment?

A second-order assignment specifies a task which requires the students to analyse a text utilising a specific framework. Let’s say the task specification for an applied linguistic module is:

Task: Write a linguistic report (R) that analyses a text (t) using X’s framework (F).

This can be represented in mathematical notation as follows:

R[F(t)] (second-order)

Note this is a second-order task. The text (t) could be a written or spoken text that is changed each semester. The framework (F) is some linguistic theory or model, studied in class, which needs to be applied to the text in order to complete the task.

A second-order task makes the generation of passable answers by AI software much more difficult to achieve since it requires understanding of the framework (F) and its application to the text (t) – a two-step process.

A first-order task would look like this:

Write a linguistic report (R) that analyses a text (t).

R[t] (first-order)

Here the task specification merely asks for a report analysing a text without any particular framework being used. This first-order task is eminently more achievable for AI software and hence increases the risk that a student may just bypass the writing and utilise AI software.

A zeroth-order task would simply specify a report to be written. I trust that no tutor would ever set such a task!

Write a linguistic report.

R   (zeroth-order)

Example Task

Here is a hypothetical task that might be set as an assignment for a module on discourse analysis:

A: Write a report that analyses a meme using Shifman’s (2013) framework.

Here the meme is the text (t) and Shifman is an academic who published his framework (F) for meme analysis in 2013.

We could supply the meme to be analysed for the students but I think allowing them to choose their own meme would also be okay since they are unlikely to pick a meme which the AI software is familiar with.

However, if the framework and the text are well-known it may be possible for the AI software to generate a passable report based on previous exemplars. For example:

B: Write a report that analyses Juliet and Romeo using Halliday and Hasan’s (1976) cohesion theory.

The popularity of Shakespeare’s plays and the extensive documentation of Halliday and Hasan’s framework means this task is simpler for AI technology, so we need to guard against this.

Testing ChatGPT

In order to demonstrate ChatGPT’s capabilities, but also to show how second-order assignments are problematic for it, I inputted the two example tasks above to ChatGPT and asked the software to generate 500-word reports for each. Testing AI software with the tasks we write is the second stage in developing robust assignments.

In the first case below – analysing a meme using Shifman’s framework – the software has failed miserably. While the report is coherent and well structured, the contents completely misunderstand Shifman’s framework. It assumes the elements of the framework are ‘meme, community and culture’ whereas in fact the framework specifies ‘content, form and stance’. The framework has been applied to a meme but the origin of the meme is not clear. It is only discussed in very general terms. The AI software has probably invented the meme as it has a tendency to do.

A: I want you to write a 500-word report that analyses a meme using Shifman’s (2013) framework.
ChatGPT output:
Memes have become an integral part of online culture, and as such, they have been subject to a significant amount of analysis and study. One framework that has been used to analyze memes is Shifman’s (2013) framework, which includes three main elements: the meme, the community, and the culture.

The first element of Shifman’s framework is the meme itself. A meme is a cultural unit that is passed from one person to another, typically through the internet. It can be a word, phrase, image, or video that is meant to be humorous or satirical. In the case of the meme being analyzed, it is an image of a person with the caption “I’m not a scientist, but I play one on TV.” The image is meant to be humorous and satirical, as it implies that the person in the image is not a scientist but is pretending to be one.

The second element of Shifman’s framework is the community. Memes are often created and shared by specific online communities, such as Reddit, 4chan, and 9gag. These communities have their own norms and values, which can influence the way that memes are created and shared. In the case of the meme being analyzed, it is likely that it was created and shared by a community of people who are interested in science and technology.

The third element of Shifman’s framework is culture. Memes are a reflection of the culture in which they are created and shared. They can be used to comment on current events, social issues, and popular culture. In the case of the meme being analyzed, it is likely that it is a commentary on the role of scientists in society. The image and caption suggest that scientists are not always taken seriously, and that some people might not even know what a scientist is.

In conclusion, Shifman’s (2013) framework is a useful tool for analyzing memes. It allows researchers to understand the meme itself, the community that created and shared it, and the culture in which it exists. In the case of the meme being analyzed, it is an image with the caption “I’m not a scientist, but I play one on TV” which is meant to be humorous and satirical and likely created by a community of people who are interested in science and technology, and it is a commentary on the role of scientists in society and how they are not always taken seriously.
fail!

In the B sample below – analyse Juliet and Romeo using the cohesion framework – the software has been able to generate a reasonable report which is passable. The report notes three features of the cohesion framework: ‘lexical, referential and conjunctive cohesion’ which are all valid aspects of the framework, although there are other features. The length of the report may have limited ChatGPT’s selection here. The AI applies these features to the text of Juliet and Romeo although the samples chosen are very general – using words such as ‘love, death’ but also simple conjunctions such as ‘but, and’.

B: I want you to write a 500-word report that analyses Juliet and Romeo using Halliday and Hasan’s (1976) cohesion theory.
ChatGPT output:
Halliday and Hasan’s (1976) cohesion theory is a linguistic tool that can be used to analyze the way that words and phrases in a text are related to one another. This theory can be applied to literary texts such as Romeo and Juliet, a play written by William Shakespeare. The play is a tragic love story of two teenagers who fall in love despite the longstanding feud between their families, the Montagues and the Capulets.

One of the key concepts in cohesion theory is lexical cohesion, which refers to the repetition of words or phrases within a text. In Romeo and Juliet, lexical cohesion is used to create a sense of unity and continuity throughout the play. For example, the play is full of words and phrases that are repeated, such as “love” and “death.” These words are repeated throughout the play, often in different forms, to create a sense of unity and continuity.

Another key concept in cohesion theory is referential cohesion, which refers to the way that words and phrases in a text are related to one another through reference. In Romeo and Juliet, referential cohesion is used to create a sense of continuity and connection between different scenes and characters. For example, the play is full of references to the feud between the Montagues and Capulets, which serves to connect different scenes and characters.

Lastly, Halliday and Hasan’s cohesion theory also includes the concept of conjunctive cohesion, which refers to the use of conjunctions, adverbials, and other cohesive devices to link clauses and sentences. In Romeo and Juliet, conjunctive cohesion is used to create a sense of continuity and connection between different scenes and characters. For example, the play is full of conjunctions such as “but,” “and,” and “or,” which are used to link clauses and sentences together.

In conclusion, Halliday and Hasan’s (1976) cohesion theory is a useful tool for analyzing the way that words and phrases in a text are related to one another. In Romeo and Juliet, lexical, referential, and conjunctive cohesion are used to create a sense of unity and continuity throughout the play. The play is full of words and phrases that are repeated, such as “love” and “death” and references to the feud between the Montagues and Capulets, which serves to connect different scenes and characters and use of conjunctions, adverbials, and other cohesive devices to link clauses and sentences. All these cohesive devices help to create an emotional impact and make the story more engaging for the audience.
pass

In the real world, the task specification would be longer. Typically we might specify 1,000 word reports for first-year undergraduates, and students would be expected to reference secondary sources to support the analysis. However the output of the second report is surprisingly good and could form the basis of a report which a student might easily embellish.

Summary of advice

  • Set second-order tasks
  • Run the task through ChatGPT or AI software before releasing to students

Halliday, M., & Hasan, R. (2013). Cohesion in English. Routledge.

Shifman, L. (2013). Memes in a Digital World: Reconciling with a Conceptual Troublemaker. Journal of Computer-mediated Communication, 18(3), 362-377.