# AI before ChatGPT: tracing the ancestry of large language models Will Berard - University of Roehampton [Watch the talk](https://youtu.be/LbUWJUz6GA4?) [This article as a pdf](https://www.hypernoetic.net/Presentations/UR+AI+PGR+Conf+2026/Berard%2C+2026+-+AI+Before+ChatGPT.pdf) *Artificial Intelligence in Research and Practice: An Interdisciplinary PGR Conference* - 17 June 2026 >[!note] Author's bio Will Berard PGCE MSc MEng is a PhD researcher at the University of Roehampton's School of Education. Trained in computer science in the early 2000s, Will worked as a software engineer for more than a decade, before training to teach Computing and Computer Science in 2021. His doctoral research inquires into higher education faculty attitudes to student use of generative AI in assessment. He is a member of the British Computing Society and the Chartered College of Teaching. ## Abstract *This conference paper retraces the lineage of generative artificial intelligence, a technology which, for many in academia, only came into our awareness with the release of ChatGPT, on the last day of November 2022 [@openaiIntroducingChatGPT2022]. It touches upon the key developments, first in 'AI' as 'deep' machine learning through artificial neural networks (ANNs), then in the large language models (LLMs) which underpin much of today's 'generative' AI, up to 2022. In showcasing underappreciated aspects of AI history, which speak to the intrinsic strengths, limitations and risks of the technology, this paper concludes with recommendations for our choices in its adoption and our appraisal of its performance.* ***This text contains no AI-generated content.*** ## How 'artificial intelligence' came to mean 'deep learning' "Artificial Intelligence" is not a technical term of art: over time, the set of target problems to which it refers has changed, and so have the techniques to solve them. When John McCarthy coined the phrase in 1955 to name a conference [@mccarthyProposalDartmouthSummer1955], this was by, his own admission, already a *branding* effort, to distinguish his work from the new field of *cybernetics* associated with his rival Norbert Wiener [@mccarthy1996defending]. Before the birth of the phrase, in 1950, Alan Turing [-@turing1950turing] writes of *Thinking Machines*, and it is in this text that he proposes his now famous *imitation game*. A game, not a test, nevertheless quickly operationalised: before even the coinage of the phrase artificial intelligence", a metric for success in the field was the successful deception of an individual, by producing text plausibly attributable to a human author. The technologies we now call AI are based on artificial neural networks (ANN), an idea first put forward as mathematics in 1943 [@mccullochLogicalCalculusIdeas1943], and first implemented in 1958, in Psychologist Frank Rosenblatt's *Perceptron* [@rosenblattPerceptronProbabilisticModel1958]. ANNs dominate in the sixties, then fall by the wayside by the end of the decade, as non-trivial problems require modelling a number of connections out of reach of the compute of the era [@minskyPerceptronsIntroductionComputational1969]. It is the first of several "AI Winters" of the twentieth century, when research in the field stalls. This one came with a change of dominant paradigm, in favour of machine learning systems based on symbolic representation of explainable rules. ANNs come back in the eighties, with two significant advances. First is the invention of a linear algebra method to train them more efficiently: *backpropagation*. Geoffrey Hinton's work on backpropagation [@rumelhartLearningRepresentationsBackpropagating1986] earned him the moniker of 'Godfather of AI', and, in 2024, a Nobel prize in Physics, along with John Hopfield, for work done almost forty years before [@ackleyLearningAlgorithmBoltzmann1985]. The year before his Nobel prize, Hinton resigned from Google Brain so he could speak freely of his grave concerns with the risks of the technology. This, however, has not helped him being heard. The other milestone in the 80s and 90s is the work of Yann LeCun on convolution neural networks, which will lead to ANN seeing their first industrial application: optical character recognition for mail sorting and cheque processing [@lecunHandwrittenDigitRecognition1989;@lecunLearningAlgorithmsClassification1995;@lecunGradientbasedLearningApplied1998]. LeCun went on to become head of AI at Meta, leaving the company in 2025. From recognising characters to recognising objects, machine vision has long been a prototypical AI problem, and it is in this field again that the next decisive milestone was laid, in the 2012 edition of *ImageNet*'s Large Scale Visual Recognition Challenge [@russakovskyImageNetLargeScale2015], a machine vision competition based on the annotated dataset *ImageNet*, the winning team, which featured Hinton and his PhD student Ilya Sutskever, did not use any new algorithms, their innovation was to run them, heavily parallelised, on re-purposed graphics processing units (GPUs) [@krizhevskyImageNetClassificationDeep2012]. Sutskever was one of the founders of OpenAI, resigning in 2024. The year before, along with other executives concerned at the company's leadership, he had tried and failed to oust Sam Altman (Sigal, 2024). Interestingly, *ImageNet*, as a dataset, already featured some of the ethical issues that still plague generative AI a decade later: questionable labour practices to annotate the images, and a resulting annotation that reinforces existing societal biases [@denton2021genealogy]. Preparing a dataset for machine learning involves invisible, but non-neutral decisions taking place even before training. ![[ImageNet Biais - Loser (Paglen, n.d.).png]] (ImageNet, in Paglen, n.d.) A better known milestone of this era is the victory of DeepMind's *AlphaGo*, in 2016, over Korean Go grandmaster Lee Sedol, mirroring that, ten years prior, of IBM's *Deep Blue* against Gary Kasparov at chess. Go had long been the prototypical *intractable* problem, for which it is not possible to rely on the brute force exploration of possible game states that was instrumental to *Deep Blue*'s win. *AlphaGo*'s victory prompted the idea that, if *one* intractable problem can be solved by ANNs, then surely, with enough training data and compute infrastructure, *all of them could*; this marked a paradigm shift in algorithmic decision making from causality to data association [@wangWhereDoesAlphaGo2016]. Training artificial neural networks, through backpropagation, on GPUs, allows them to overcome the limits perceptrons hit at the end of the 1960s. Neural nets can be much larger, have more 'layers' of depth. What we have been calling 'artificial intelligence' since the 2010s is better described as 'deep learning'. Deep learning is a fine technology, with too many applications to mention; at its core, its function is to *find patterns* in data - either patterns the systems have been trained to recognise, or patterns *latent* in the data. It is an idea so old that it has been around us since the late 80s to automate the processing of snail mail and cheque cashing, two areas which do not exactly scream 'cutting edge'. The only difference is that now we have the computing power and the datasets to make it work - especially the Big Tech behemoths. ## A genealogy of the AI chatbot The semantic plasticity of the term AI means we're putting in the same category deep learning and the latest wave of AI, heralded by ChatGPT [@openaiIntroducingChatGPT2022], the 'generative' AI of large language models (LLMs). Like machine vision, natural language processing (NLP) has long been an important branch of "AI", and in this domain, chatbots are important *proofs of concept*. First amongst these, in impact if not date, is Joseph Weizenbaum's ELIZA, particularly convincing in its DOCTOR personality of Rogerian psychotherapist, bouncing the user's statements back as questions, and, when it couldn't, answering "and how does that make you feel?".[@weizenbaum1966eliza]. "Weizenbaum was shocked to discover that so many users were taking his program seriously and that they were sharing their most private thoughts with the machine." [@oreganElizaProgram2018 p. 119]. Weizenbaum went on to become a "leading critic of AI research" (_ibid._). ELIZA would use syntactical rules, which could be audited and modified. Today's chatbots, though, are backed by large language models. LLMs are deep learning systems that have found latent patterns in their training data, and reproduce these, by continuing a' context' - the text input. Text for LLMs, is a series of *tokens* (words, parts of words, as well as punctuation and control characters), and each token is represented by a vector. These *"vector embeddings*" are arrived at through training on a guess-the-missing-word exercise. In practice each vector is a lists of numbers that can be thought of as coordinates in a hyperdimensional space [@mikolovDistributedRepresentationsWords2013]. These vectors hold a form of meaning; as in the example: $vec(queen) - vec(woman) \approx vec(king) -vec(man)$ "'Queen' is to 'woman' as 'king' is to 'man'": in the 'latent space' of meaning (as represented in the training set), the relative position of 'man' and 'king' mirrors that of 'woman' and 'queen'. Meaning here is *differential* - vectors in themselves hold none, only *relationships* between vectors do, and even then, they only reflecting the statistics of the training set. A similar near equivalence often occurs whereby man is to woman as doctor is to nurse [@nissimFairBetterSensational2020]. Embeddings only represent *standing* - decontextualised meaning [@griceUtterersMeaningSentenceMeaning1968;@quineWordObjectNew2013]. There is only one vector for the word "bank", and it somehow captures its meanings both as riverside, financial institution, or the tilting of an aeroplane. Meaning *in context* can then be modelled in different ways; for ChatGPT and many other chatbots, this is achieved with a *transformer architecture*, pioneered by OpenAI's GPT-1. A 'self-attention mechanism', adjusts the weight the contribution of each of the context's tokens, and each of their vectors' position is adjusted to better reflect their meaning, given the surrounding words [@vaswaniAttentionAllYou2017]. But the function of these systems remains predicting the next likely token based on the context, because this is all they are trained to do. LLMs *model* meaning - as encountered in their training and 'fine-tuning', but do not *understand*. They have no moral compass, no reference to ground truths, no model of the world beyond their training data, and their output is devoid of communicative intent [@benderClimbingNLUMeaning2020]. I cannot stress enough how important that is. Still, this approach has yielded more and more impressive results, first by separating generalist 'pre-training' from application-specialised 'fine-tuning' [@radford2018improving], then by scaling size of models and training sets to a level only accessible to the handful of organisations developing the technology, with, for OpenAI, GPT-2 [@radford2019language], then GPT-3 [@brown2020language]. This caused, at the time, critical voices to be raised, most notably these of Emily M. Bender and colleagues [-@benderDangersStochasticParrots2021], whose paper coined the term "stochastic parrot". They argue that the emergent capabilities of ever-larger models are nothing but the reproduction of pattern latents in the data, and that seeking ever-larger training sets make them *de-facto* impossible to review, curate, or even obtain ethically. ## Conclusion Just like deep learning, there are many specialised applications to LLMs, some of which can run on a typical laptop: "utility does not require scale" [@varoquauxHypeSustainabilityPrice2024 p. 3]. The focus on scale of the industry has put it on a trajectory that is entirely unsustainable, financially as well as environmentally. We should be more worried about this than we are. The trend, since ChatGPT, to package LLMs as anthropomorphic agents also has concerning consequences; 2025 saw the coinage of the phrase "chatbot-induced psychosis", but we have to wonder if there may be a sub-clinical flavour of the syndrome, where one does not lose contact with the whole of reality, just the part that concerns a realistic assessment of the capacities of AI systems. GPT-3 had been out, quietly, for three years, and applied with only mild excitement across many domain-specific uses, but it took the 'low-key research preview' of ChatGPT to fire the starting gun on generative AI. This is no accident. As far back as the mid-sixties with ELIZA, we have established the tendency of humans to ascribe more intelligence, understanding, agency, and a host of other human features to computer systems that interact in human language [@oreganElizaProgram2018]. It is called the ELIZA effect. We should deliberately compensate for this, because generative AI pushes certain buttons in the very wiring of our nervous systems. Many put the roots of our species spirituality in animism and *participation mystique*: the ascription of agency to inanimate objects [@guthrieanimal]. ![[I AM ALIVE.png|500]] I would recommend than in all our dealings with AI, we remember we will have a natural tendency to see more where there is, in reality, less, and correct our appraisal accordingly. It would also help to be precise in our language and distinguish deep-learning from LLMs, and, within these, specialised, often local, ones from hyperscale, 'frontier' models. The way the discourse has been framed is not helping: if McCarthy had not come up with this buzzword, 'artificial intelligence' may sound as odd as 'artificial strength'. Like for strength, there is an subjective, phenomenological aspect to intelligence. There is an intent behind its application, as well as ethical considerations to it. There is the joy of solving a problem, which cultivates a desire to solve more. This is not just a philosophical objection: forgetting those facts may lead us expect more of *intelligence-as-patterns* than it can deliver. Since Turing, we have equated intelligence with the sole outwards capability to solve problems, but few among us would we call a forklift 'strong', because it can lift boxes. ## References Ackley, D.H., Hinton, G.E. and Sejnowski, T.J. (1985) ‘A Learning Algorithm for Boltzmann Machines’, _Cognitive Science_, 9(1), pp. 147–169. Available at: [https://doi.org/10.1207/s15516709cog0901_7](https://doi.org/10.1207/s15516709cog0901_7). Bender, E.M. _et al._ (2021) ‘On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜’, _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. New York, NY, USA: Association for Computing Machinery (FAccT ’21), pp. 610–623. Available at: [https://doi.org/10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922). Bender, E.M. and Koller, A. (2020) ‘Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data’, in D. Jurafsky et al. (eds) _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. _ACL 2020_, Online: Association for Computational Linguistics, pp. 5185–5198. Available at: [https://doi.org/10.18653/v1/2020.acl-main.463](https://doi.org/10.18653/v1/2020.acl-main.463). Brown, T. _et al._ (2020) ‘Language models are few-shot learners’, _Advances in neural information processing systems_, 33, pp. 1877–1901. Denton, E. _et al._ (2021) ‘On the genealogy of machine learning datasets: A critical history of ImageNet’, _Big Data & Society_, 8(2), p. 20539517211035955. Grice, H.P. (1968) ‘Utterer’s Meaning, Sentence-Meaning, and Word-Meaning’, _Foundations of Language_, 4(3), pp. 225–242. Guthrie, S.E. (2002) ‘Animal animism: Evolutionary roots of religious cognition’, in Ilkka Pyysiäinen and Veikko Anttonen, _Current approaches in the cognitive science of religion_, pp. 38–67. Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) ‘ImageNet Classification with Deep Convolutional Neural Networks’, _Advances in Neural Information Processing Systems_. Curran Associates, Inc. Available at: [https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html](https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html) (Accessed: 11 August 2025). LeCun, Y. _et al._ (1989) ‘Handwritten digit recognition with a back-propagation network’, in D. Touretzky (ed.) _Advances in neural information processing systems_. Morgan-Kaufmann. Available at: [https://proceedings.neurips.cc/paper_files/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf). LeCun, Y. _et al._ (1995) ‘Learning algorithms for classification: A comparison on handwritten digit recognition’. Available at: [https://www.semanticscholar.org/paper/Learning-algorithms-for-classification%3A-A-on-digit-LeCun-Jackel/842dd6d0f4b72ce0e8f3ac8e6861637c1f4645ea](https://www.semanticscholar.org/paper/Learning-algorithms-for-classification%3A-A-on-digit-LeCun-Jackel/842dd6d0f4b72ce0e8f3ac8e6861637c1f4645ea) (Accessed: 11 August 2025). Lecun, Y. _et al._ (1998) ‘Gradient-based learning applied to document recognition’, _Proceedings of the IEEE_, 86(11), pp. 2278–2324. Available at: [https://doi.org/10.1109/5.726791](https://doi.org/10.1109/5.726791). McCarthy, J. (1996) ‘Defending AI research: a collection of essays and reviews; McCarthy, J., ed’, _Center for the Study of Language and Information: Stanford, CA, USA_ [Preprint]. McCulloch, W.S. and Pitts, W. (1943) ‘A logical calculus of the ideas immanent in nervous activity’, _The bulletin of mathematical biophysics_, 5(4), pp. 115–133. Available at: [https://doi.org/10.1007/BF02478259](https://doi.org/10.1007/BF02478259). Mikolov, T. _et al._ (2013) ‘Distributed Representations of Words and Phrases and their Compositionality’. arXiv. Available at: [https://doi.org/10.48550/arXiv.1310.4546](https://doi.org/10.48550/arXiv.1310.4546). Minsky, M. and Papert, S.A. (1969) _Perceptrons: An Introduction to Computational Geometry_. The MIT Press. Available at: [https://doi.org/10.7551/mitpress/11301.001.0001](https://doi.org/10.7551/mitpress/11301.001.0001). Nissim, M., van Noord, R. and van der Goot, R. (2020) ‘Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor’, _Computational Linguistics_, 46(2), pp. 487–497. Available at: [https://doi.org/10.1162/coli_a_00379](https://doi.org/10.1162/coli_a_00379). OpenAI (2022) _Introducing ChatGPT_, _OpenAI_. Available at: [https://openai.com/index/chatgpt/](https://openai.com/index/chatgpt/) (Accessed: 14 April 2025). O’Regan, G. (2018) ‘Eliza Program’, in G. O’Regan (ed.) _The Innovation in Computing Companion: A Compendium of Select, Pivotal Inventions_. Cham: Springer International Publishing, pp. 119–122. Available at: [https://doi.org/10.1007/978-3-030-02619-6_24](https://doi.org/10.1007/978-3-030-02619-6_24). Quine, W.V.O. (2013) _Word and Object, new edition_. MIT Press. Radford, A. _et al._ (2018) ‘Improving language understanding by generative pre-training’. Radford, A. _et al._ (2019) ‘Language models are unsupervised multitask learners’, _OpenAI blog_, 1(8), p. 9. Rosenblatt, F. (1958) ‘The perceptron: A probabilistic model for information storage and organization in the brain.’, _Psychological Review_, 65(6), pp. 386–408. Available at: [https://doi.org/10.1037/h0042519](https://doi.org/10.1037/h0042519). Rumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986) ‘Learning representations by back-propagating errors’, _Nature_, 323(6088), pp. 533–536. Available at: [https://doi.org/10.1038/323533a0](https://doi.org/10.1038/323533a0). Russakovsky, O. _et al._ (2015) ‘ImageNet Large Scale Visual Recognition Challenge’, _International Journal of Computer Vision_, 115(3), pp. 211–252. Available at: [https://doi.org/10.1007/s11263-015-0816-y](https://doi.org/10.1007/s11263-015-0816-y). Sigal, S. (2024) ‘Why the OpenAI superalignment team in charge of AI safety imploded - …’, _Vox_, 18 May. Available at: [https://archive.ph/FEpq9](https://archive.ph/FEpq9) (Accessed: 10 May 2026). Trevor Paglen (no date) _ImageNet Roulette_, _paglen.studio_. Available at: [https://paglen.studio/2020/04/29/imagenet-roulette/](https://paglen.studio/2020/04/29/imagenet-roulette/) (Accessed: 10 May 2026). Turing, A. (1950) ‘Computing machinery and intelligence.’, _Mind_, 59(236), pp. 433–60. Varoquaux, G., Luccioni, A.S. and Whittaker, M. (2024) ‘Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI’. arXiv. Available at: [https://doi.org/10.48550/arXiv.2409.14160](https://doi.org/10.48550/arXiv.2409.14160). Vaswani, A. _et al._ (2017) ‘Attention is All you Need’, _Advances in Neural Information Processing Systems_. Curran Associates, Inc. Available at: [https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html) (Accessed: 8 June 2025). Wang, F.-Y. _et al._ (2016) ‘Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond’, _IEEE/CAA Journal of Automatica Sinica_, 3(2), pp. 113–120. Available at: [https://doi.org/10.1109/JAS.2016.7471613](https://doi.org/10.1109/JAS.2016.7471613). Weizenbaum, J. (1966) ‘ELIZA—a computer program for the study of natural language communication between man and machine’, _Communications of the ACM_, 9(1), pp. 36–45.