This is a transcript (slightly edited and shortened) of a talk I gave as part of Edinburgh’s Centre for Research in Digital Education Seminars Series. The full recording is here
The explosion of generative AI in 2022 has been felt keenly in education. Whether or not we are facing a truly disruptive set of circumstances is open to debate, but our ability to grasp and interpret the problem at hand has most definitely been challenged. While the influential critical literature around AI offers a valuable basis to build upon (Crawford, 2021; Parisi, 2019), there is a generalised sense that innovation is outpacing our ability to theorise.
Technological innovation has reached a point where the extractive sociotechnical apparatus of AI does not only capture value, understood as labour and affect (Perrotta et al., 2022). AI can now successfully exploit such value to produce human-like and, in some scenarios, more-than-human knowledge, conveyed with unprecedented communicative competence.
The issue of knowledge is therefore central to the discussion: what is generative AI doing to knowledge, if it’s doing anything at all? The question is highly theoretical, but it has pragmatic implications in educational assessment, and indeed in almost any sphere of human activity.
In this talk, I am interested in the interpretative possibilities at our disposal. I tentatively draw on Foucault’s work on the historical analysis of knowledge (Foucault, 1966) and, in particular, I pay attention to his use of Cervantes’ Don Quixote as a case study to examine the changes that occurred in knowledge in the 15th and 16th Centuries, as language began to break relationships of similitude and resemblance with reality. Foucault argued that Don Quixote is a tragic figure who has not moved on with the times – he is not aligned with the change in the epistemic landscape, where the similarities between things are no longer a source of knowledge but a source of confusion to be unpicked and exposed. In this new, confusing reality, Don Quixote is locked in a relentless quest, relying on the codifications of chivalric fiction to confirm his own very existence: ‘He must endow with reality the signs-without-content of the narrative.’
The talk engages with the thesis – without committing to it – that we are entering a new epistemic condition where generative AI rearranges our relationship with language and meaning along similar lines. In this sense, generative AI is the culmination of what Foucault called the ‘Leibnizian project of establishing a mathematics of qualitative orders’, based on an endlessly re-negotiable distance that separates weak probabilities and strong certainties.
The overarching provocation is that Don Quixote is a fitting metaphor for what we are experiencing in education in the current historical moment: epistemic confusion, simulated competence, and automated extravagance.
****
I don’t think I am the only one who feels discombobulated. Over the past six months my response to the constant stream of news and developments about generative AI and education has shifted from scepticism – a sense that we have been here before – to anxiety caused by FOMO and emotional contagion – everybody was getting very excited and worried about generative AI, maybe I should feel that too – to a sense of world-weariness at the realisation that we are sleepwalking into a massive social experiment meant to test the viability of a much worse version of reality which promises some efficiencies in exchange for a massive over expenditure in terms of vigilance and moderation.
The concept of dual use can be used to frame this problem, before we get into the more speculative discussion about Don Quixote and Foucault – Dual Use is the notion that some technologies are more problematic than others because their pro-social implementations can be easily subverted and used for bad purposes by bad actors. Is a fundamentally utilitarian and binary view of technological innovation. It has problems but it is helpful heuristically – nuclear energy is the classic example of dual use because the same expertise and to an extent the same infrastructure used to create an alternative to fossil fuels (good) can be used for nuclear weaponry (bad).
The notion of dual use can be thought of as a dynamic relationship between morally classifiable uses (good and bad), plus number of actors involved and the level at which they operate – from the societal level to the individual level. In the case of atomic energy, we have a relatively manageable subset of good and bad uses (alternative energy source vs WMD and a handful of others?), and a relatively manageable subset of actors involved in that dynamic who can be held responsible for a particular bad use: sovereign states, a handful of large private entities, intelligence services perhaps. The subset can be expanded considerably depending on one’s theoretical leanings, you could use an actor-network perspective where agency is so distributed and horizontal that the very notion of accountability becomes diluted to the point of irrelevance.
Now let’s apply the same logic to generative AI, and the situation changes dramatically:
A) good and bad uses grow exponentially.
B) Good uses and bad uses scale up in completely unpredictable ways – and this has something to do with the way platform logics and network effects get entangled with uses.
C) The number of actors who can be held responsible for bad uses also grows exponentially.
D) Good uses and bad uses are unequally distributed in terms of value, where value is understood in a political-economic way following an extractive logic: there can be high value good uses, low value good uses, high value bad uses and bad value bad uses.
E) Accountability also becomes multifaceted and fragmented all the way down to a very granular level of agency and conduct – is a company using AI-generated propaganda (high value bad use) as accountable as a child using ChatGPT to cheat at school (low value bad use)?
In sum, we are rapidly reaching a point where the simple quantification and classification of good and bad uses of AI becomes so problematic that it calls into question our entire system of institutions and our political and legal mechanisms of checks and balances. Furthermore, none of this is the result of democratic discourse of deliberation – it’s instead a corporate imposition on a planetary scale.
The dual use framework is thus helpful to problematize and disturb the dominant narrative of technological disruption that is being used to normalise generative AI. In education, there is a popular argument according to which generative AI is not much different than calculators, with a similar augmentative relationship between students and technology that is now emerging but it is being opposed by obscurantist and conservative forces. The analogy however breaks down when we consider the differences between dual use scenarios that I sketched earlier. We are definitely not talking about calculators here! We are dealing with something qualitatively and quantitatively different and I think acknowledging this fact will help elevate the discussion, especially in education – thus challenging the ‘instrumental definition’ of AI and technology more broadly.
Andrew Feenberg summarises this definition as follows (Feenberg, 1991): ‘The instrumentalist theory offers the most widely accepted view of technology. It is based on the common sense idea that technologies are ‘tools’ standing ready to serve the purposes of users.’
With all this dual-use stuff in mind, let’s move to Don Quixote and Foucault – especially the Foucault of the Order of Things, a book concerned with the analysis of systems of knowledge through the archaeological method of analysis which Foucault then refined a few years later.
My thesis is that the dual use complexity of AI is the manifestation of a broader epistemological shift – not a disruption in the techno-romantic sense but a discontinuity, which comes again from Foucault, who viewed the history of knowledge as a sequence of discontinuities where particular discourses (psychiatry, madness, psychology) emerged in a particular moment as relatively cohesive systems of meaning connected to other systems of meanings that came before. What we know and how see the world now was already present in some embryonic form before:
“everything that is formulated in discourse was already articulated in that semi-silence that precedes it, which continues to run obstinately beneath it, but which it covers and silences”
What I am trying to achieve in this talk is to historicise the moment we are experiencing thanks to generative AI, and I truly believe this is a disruptive moment or at very least a moment of discontinuity. To historicise means that nothing suddenly appears on the stage of history completely unannounced, but there are always roots and paths that can be re-traced or dug-up in a geological or indeed archaeological way. With this, we get to Cervantes and Don Quixote.
Cervantes’ Don Quixote is a satirical novel that ridicules romantic novels of the 15th and 16th centuries, which were the dominant form of storytelling at the time. The novel was highly innovative and influential, and it has been the object of a great deal of analysis and exegesis over the past three centuries. Like all great works of literature, it has multiple levels of interpretation. from a purely thematic point of view the novel is a cautionary tale about the dangers of bad fiction and escapism, which seems as current now as it was back then: the main character, Don Quixote de la Mancha, is an old country gentleman who, after having spent a good part of his adult life reading chivalric novels, makes the unilateral decision that is living in one. From a critical humanities perspective, the novel is one of the earliest examples of ironic deconstruction of established styles and conventions – something which many years later came to be associated with modernity and post-modernity. For romantic philosophers and authors, Don Quixote was a tragic hero who fought a hopeless yet brave battle against a mediocre and trivial reality, using his vivid imagination and a firm moral code as his primary weapons.
Even if we stop at this, relatively superficial, level of analysis, and in particular at the notion of Don Quixote as a figure that operates in a liminal space, between fiction and reality, I found the work to be helpful as a metaphor to make sense of the current moment of confusion in education – and more broadly.
People more qualified than me in educational assessment have already argued very convincingly that generative AI is simply shining a light on the profound inadequacy of the current paradigm, which has gradually, over a relatively long period, decoupled itself from the pedagogical and social values that were supposed to inform it. Decades of excessive dependency on formulaic writing and on trivialised forms of factual learning; an uncritical proliferation of stock-standards rubrics and criteria, quite literally exportable from one student cohort to another, from one age group to another, and even from one education system to another; years of curricular standardisation, quantification and accountability. These trends have laid the foundations on which ChatGPT now stands.
The only reason why generative AI is so good at writing essays, is because in large part essays have become formulaic and standardised – both in terms of form and content, and the same could be said about other forms of ‘traditional’ assessment in secondary and tertiary education.
In this sense, trying to defend the status quo increasingly feels like a quixotic endeavour, because we were already living in a fictionalised version of education – generative AI is simply exposing the ruse. In that regard, I agree with the argument that chatGPT can be a good thing for educational assessment which could trigger a much-needed process of reform. That reform would certainly require an acknowledgement of the fundamentally tragic and inadequate nature of current education, but also (and this is where I depart from some of the more enthusiastic mainstream accounts) an awareness that generative AI may not be the messianic solution that everyone was eagerly awaiting. Going back to the Quixotic metaphor, it would be comparable to Don Quixote’s reaction when, in his famous first adventure, he fights windmills thinking they are giants, and when he finally gives in to the realisation of what is actually happening, concludes that a magician must have turned giants into windmills. In doing so, he does not trade delusion for reality but articulates instead an explanation that allows him to cling to his psychotic imagination. As a result, he falls even deeper into his delusions reinforcing his allegiance to the fictionalised reality he chose to inhabit.
Don Quixote is duty bound to his tales of heroic chivalry which no longer has a relationship of resemblance with reality – the fact that the signs of the narrative no longer resemble the visible world causes his tragedy, as he relentlessly needs to refer to the book to confirm the conditions of possibility of his very existence: ‘He must endow with reality the signs-without-content of the narrative.’
Generative AI exposes the artifice of contemporary education because the formulaic approach to knowledge production and transmission that plagues assessment (e.g. a ‘critical’ essay with a neat introduction, a pros/cons analysis and a ‘balanced’ conclusion) became hard-coded in its algorithmic logic following its autoregressive training, which enables it to predict words based on what came before (the phrase ‘what came before’ is of the essence here). ChatGPT is therefore only a mirror held up to us and cannot help to magically rethink assessment, because it has no clue how. Relying on Generative AI as an augmentative technology in educational assessment (a super-powered calculator) runs the risk of being a delusional effort that reinforces the legitimacy of the status quo.
Put differently, integrating generative AI into assessment sounds innovative, futuristic and trendy, but clearly the opportunity lies in its more modest function as a mere stimulus for the design of alternative forms of assessment. Other than that, generative AI remains a very effective but also very limited tool to automate and streamline the status quo and, as such, it is not conducive to innovation but to conservatism and ossification. So clearly we must look elsewhere for inspiration – perhaps the key lies in rediscovering the oral dimension of assessment, or other ways of evaluating performance and progress that do not rely so heavily on codified text and language.
References
Crawford, K. (2021). Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press.
Feenberg, A. (1991). Critical theory of technology , publisher = Oxford University Press New York. In.
Foucault, M. (1966). The order of things. Routledge.
Parisi, L. (2019). Critical computation: Digital automata and general artificial thinking. Theory, Culture & Society, 36(2), 89-121 , ISSN = 0263-2764.
Perrotta, C., Selwyn, N., & Ewin, C. (2022). Artificial intelligence and the affective labour of understanding: The intimate moderation of a language model. New Media & Society, 14614448221075296.
automatED 
Leave a comment