The AI Mirage: The Mechanization of Thought

In the previous two posts, I argued that artificial "intelligence" based on language modeling is not close to becoming "intelligent" in any kind of human sense, and the technology's prospects for becoming such are not promising. They perform or imitate the product of intelligence, namely language. This is drastically different from the stuff of human thought, which is not reducible to language.

In this post I will explore the question: what happens if we deny this? What if we claim, as AI leaders wishfully do, that the only difference between their models and our minds is one of degree, a gap which is rapidly dwindling to nothing if it hasn't vanished already?

An information age

Ever since I picked up Against the Machine last year, I've been exploring the ways that the scientific revolution, technological advance, and the onset of (post-)modernity have influenced how we think and speak about ourselves and the world around us. As Sherry Turkle says, technology makes us forget what we know about life. And as Neil Postman warned, technology can lead us to redefine fundamental concepts in our culture without our even noticing the shift. What might chatbots lead us to forget or redefine about ourselves?

The British Orthodox scholar Philip Sherrard explored these questions (for science as it was in the 90s) in depth in his powerful book, The Rape of Man and Nature, and describes the distinctively modern viewpoint in a passage I think is worth quoting at length:

Man is now regarded as a more or less autonomous entity existing as it were apart from God. He is thought of as being nothing more than an individual and finite creature, and his reason, cut off and closed to all that is above it, is regarded as the sole instrument and arbiter of his knowledge, anything that transcends the reason being treated quite simply as non-knowledge or, which is the same thing, as the sub-product of a hallucinatory and irrelevant imagination.

As for this knowledge itself, it has nothing sacred about it. It does not refer to any sacred or qualitative reality, and it does not require from man any deep inner transformation and ascesis in order to acquire it: there is no awareness any longer of the bond that links knowledge of the truth and the state of purity so intimately that there can be no knowing in any but an abstract and impersonal sense until such purity is attained. On the contrary, what is now regarded as knowledge is something entirely profane, and in so far as it refers to anything at all, it refers but to a nature that is seen as dead, alien, and purely functional and quantitative. As such, this knowledge is at the disposal of any deft investigator who will exert his brain but at the same time may be mean-spirited, selfish and given over to pursuits as self-corruptive as they are trivial. In other words, we have entered the world of modern science, with all its consequences for human and natural life.

And later:

[The shift to the Cartesian world] meant the expulsion from scientific thought of all considerations based on value, perfection, harmony, meaning, beauty, purpose, for such considerations are regarded as merely subjective and so as irrelevant to a scientific understanding of the real 'objective' world—the world of quantity, of reified geometry, of a nature that is impersonal and purely functional. . . . There is nothing that is not reduced either phenomenon (fact) or to mathematical hypothesis (or, in less polite language, fiction).

The kind of profane, purely quantitative, impersonal knowledge Sherrard describes is characteristic of the secular, scientific way we have viewed the physical world for some time now, the consequences of which Sherrard explores in the last chapter of his book. It is knowledge for control, the lifeblood of what Kingsnorth describes as the Machine, to the exclusion of "value, perfection, harmony, meaning, beauty, purpose". It is increasingly the only way we moderns can imagine the world. Even for many Christians, the traditional alternative—the sacramental view of nature as divine creation, the revelation of God's glory, made for participation in His grace—is hard to consistently keep in view in daily life, unless perhaps we find ourselves in a church or place of natural beauty. We affirm it on Sunday, but soon resume swimming with the flow of our culture as we have been trained to do.

But we have been more reluctant to apply the mechanical, secular view of things to ourselves. Following Descartes, we imagine that the human mind, or soul, has some ineffable spark not reducible to mere mechanism, not fully graspable by this sort of profane knowledge. Even if we tend to imagine people distant from us as something more like abstractions (or statistics), we don't think this way about ourselves or those close to us. This is only a lingering shadow of the deeper, richer Christian anthropology Sherrard describes, but it is still a gap in secular modernity's dominant position in our culture.

The danger of the chatbot revolution is that it threatens to make a purely mechanical, functional, informational view of the human soul, which today is mostly held by philosophers, scientists, and other academics, as universal and 'obvious' to the modern mind as the secular view of the natural world already is. Just as the practical successes of the scientific revolution made the closed, mechanistic worldview assumed by the scientific method seem sufficient for modern people, so chatbots' increasingly fluent performance of intelligence makes the reduction of thought and mind to calculation performable by a machine and of the knowable to information storable as bits seem more plausible, even appealing. When chatbots (seem to) have all the answers, why overcomplicate things by thinking for yourself? Robin Phillips, after recommending Metropolitan Anthony Bloom's book Courage to Pray to a curious friend only for him to ask ChatGPT to summarize its 25 main points instead of reading it, reflects:

On the principle that the medium is the message, the message of systems like ChatGPT, Gemini, Claude, Mistral, Jasper, Falcon, Llama, etc., is that everything is reducible to information. And, of course, once something is reduced to information, then we are in control of it—we can evaluate it based on how it conforms to our own preconceived notions, which seems to be what my friend, at least in part, did.

Just as the scientific worldview enables us to "understand", and thereby control, the physical world around us, the mechanization of thought allows us to "reduce" everything to information, the better to control it. Postman updates the old adage to our era: "To a man with a computer, everything looks like data." Every challenge we face looks like an optimization problem, an engineering project. This is the worldview that allows us to elevate, with University of Chicago economist Gary Becker, the economic mindset as "a comprehensive one that is applicable to all human behavior". To apply this mindset and its technique of cost-benefit analysis (or, as it is called today, "min-maxing") to remake fields as diverse as sports, environmental conservation, politics, health, and charitable giving. To invent bold new financial technologies like repackaging peoples' life insurance policies as investment products. To claim, as economist Joel Waldfogel did, that holiday gift-giving is "inefficient" because we don't know what our recipients want as well as they do, "wasting" $13 billion of value a year.

We speak of the present as an "information age", but I wonder if this is because we have left a prior "knowledge age" behind, as Plato's King Thamus much earlier feared that writing would reduce wisdom to mere knowledge.

Human model collapse?

The other side of the claim that language models can become conscious is the idea that conscious beings like us are simply "informational organisms" whose consciousness is of the same kind as what chatbots do. Or, as OpenAI founder Sam Altman put it, "i am a stochastic parrot, and so r u". (In other words, humans, like chatbots, are just information processors that manipulate words, reducible to their inputs and outputs) Emily Bender and Alex Hanna, in The AI Con, argue that "AI hype reduces the human condition to one of computability, quantification, and rationality." Neil Postman's prediction that the computer would invite us to see ourselves and all of nature through an informational lens was perhaps a little early, but prophetic. This metaphorical message was easier to miss when computers were clunky, pixelated home appliances best suited for spreadsheets and word processing, but seems a lot more plausible when anyone can readily converse with a chatbot capable of passing Turing's test as they would with a friend—through an electronic device in their pocket.

Despite chatbots' manifest shortcomings, futurists like Ray Kurzweil and Yuval Harari have enough confidence in their progress to flip the script on programs "catching up" to human intelligence—to rather see humans as slower, inferior, hopelessly ignorant and biased computers who need the guidance and assistance of faster, "objective", broader-minded AI to reach their full potential. To adapt a statement by Epic Games: Thinking is hard. So, instead of doing that, why not let a chatbot pretend to do it for you? These programs are programmed and trained to imitate us, and now we are making them our models of rationality—a vicious cycle indeed (remember the warnings about model collapse in the previous post?). As the philosopher Byung-chul Han remarks, "The main danger that arises from machine intelligence is that human thinking will adapt to it and itself become mechanical." Robin Phillips asks questions about this recursive imitation that should give us pause:

• As the computer becomes the symbol of true objectivity, are we at risk of losing an appreciation for the material conditions of knowing, including the role physical processes play in growing wise?
• Are we at risk of minimizing, or even despising, those traits which we do not share with machines, traits like altruism, wonder, inefficiency, empathy, vulnerability, and risk-taking?
• When we hand the management of the world over to machines, do we lose the incentive to understand the world for ourselves and pass our knowledge to the next generation?
• Could over-reliance on AI assistants lead humans to have a false confidence in the creative or intellectual products they produce under the guidance of bots, given a bias toward the machine's apparently more effective and objective solutions (i.e. "the bot must be seeing something I don't")?

Pope Leo XIV, in his encyclical Magnifica Humanitas, warns of the danger of dehumanization implicit in the pro-AI stance and the broader stream of what Postman calls "technopoly".

the pervasive technocratic paradigm in which we are immersed, and that is amplified by the digital revolution and AI, threatens to normalize an anti-human vision. In that vision, the fullness of life is equated with having more, reducing weakness, eliminating uncertainty and exerting total control. When efficiency becomes the ultimate measure of value, human beings are tempted to see themselves as a project to be optimized rather than as persons called to relationship and communion.

And later, writing on the more radical quasi-religions of technology of transhumanism and posthumanism:

If the human being is treated as something to be perfected or surpassed, it becomes easier to accept that some lives are less useful, less desirable or less worthy. In the name of progress, “necessary sacrifices” may begin to be justified, placing the burden on the most vulnerable in pursuit of a supposed optimization of the species. In this regard, the aforementioned warning of Saint Paul VI retains great foresight: indeed, scientific and technological advances, when detached from moral and social progress, end up turning against humanity. For this reason, a clear distinction must be made. It is one thing to integrate technology within a human-centered, relational vision; it is quite another to be guided by an outlook that devalues human limits and promises a purely technical form of “salvation.”

To seek to exchange our human limitations, imperfect but flexible minds, and vulnerable dependence on each other for an artificial certainty obtained by the banishment of ambiguity and reduction of intelligence to chatbots' pathetic performance of it is to gain nothing real and lose our humanity itself. The pope's mention of "progress" and "optimization of the species" chillingly reminds me of the "science" of eugenics, which resembles our society's all-out charge toward electronic imitations of thought in its readiness to reduce something great, God-given, and ultimately mysterious to something mean, simple, and optimizable. When chatbots become our ideal, they cease to be mere tools capable of serving distinctly human interests. As Bender and Hanna write of hyped-up claims that chatbots are or will soon be conscious, "when we imbue these systems with fictitious consciousness, we are implicitly devaluing what it means to be human".

Douglas Hofstadter, writing in the 70s, could have been critiquing modern machine-extruded music (which is infamously capable of aping famous musicians' style) when he hypothesized a device capable of producing music that Chopin or Bach might have written had they lived longer, dismissing such a device as "a grotesque and shameful misestimation of the depth of the human spirit". To reduce human creativity to something automatable by a machine is to turn it into something else entirely.

And so with empathy and other interpersonal skills. Joseph Weizenbaum, the creator of ELIZA who was shocked at how readily people imputed human traits to the rudimentary chabot, wrote in Computer Power and Human Reason: "There are important differences between men and machines as thinkers. I would argue that, however intelligent machines may be made to be, there are some acts of thought that ought to be attempted only by humans." Among these he includes human functions such as interpersonal respect, understanding, and love, echoing his alarm at the suggestions that ELIZA be used in psychiatric settings—suggestions that are now being fulfilled as modern chatbots are being presented as therapists. He insists (and we would be wise to listen): "respect, understanding, and love are not technical problems." Morality is not reducible to a mathematical optimization problem. Chatbots, Sherry Turkle says, don't have emotions, whatever their engineers might say. They work by simply predicting the next likely words to string into a sentence. Their performance of empathy, however superficially convincing, remains just that, nothing more.

The ocean of consciousness

You may wonder why I am being so negative. As a software engineer, I know well how rapidly computers and the software they run have advanced. How am I so sure that AI companies, with their increasingly immense resources, will not close the remaining gaps between human and machine intelligence in the coming years? In light of the stunning advances of genAI in recent years, how am I certain they are nowhere near consciousness?

Because making this claim requires doing extreme violence to the fullness of what the words "intelligence" and "conscious" mean—or used to mean, at any rate. Chatbots are only intelligent if we move the goalposts and forget most of what we as human are capable of. Neil Postman, drawing conclusions from ELIZA, wrote in the early 90s: "Human intelligence, as Weizenbaum has tried energetically to remind everyone, is not transferable. The plain fact is that humans have a unique, biologically rooted, intangible mental life which in some limited respects can be simulated by a machine but can never be duplicated. Machines cannot feel and, just as important, cannot understand." What is missing from computers' performance of human intelligence, among other things, is meaning. Postman continues, linking this with related dimensions of human intelligence: "It is meaning, not utterance, that makes mind unique. . . . As I understand it, meaning also includes those things we call feelings, experiences, and sensations that do not have to be, and sometimes cannot be, put into symbols. They 'mean' nonetheless. Without concrete symbols, a computer is merely a pile of junk." Communication via symbols (i.e. language), as I argued last time, is not thought; that the two normally accompany each other in our experience makes us vulnerable to being deceived by smooth-talking chatbots.

As I also argued last time, human intelligence is ultimately unintelligible. We know it when we see it (and can readily recognize its absence in the kinds of genAI interactions I shared two posts ago), but we can't completely define it in a way that gives us the kind of control the modern mind seeks. Consciousness, too, is a mystery; it is ultimately too vast, too deep to fully know itself in any sort of clear, explicit way. The danger of chatbots is that they can lead us to forget this and conclude there is no more to intelligence than what we can readily see.

The question is not how I know language models are not capable of real intelligence, but how anyone could sensibly believe that they are. Why should intelligence be so easily separable from our memories, cares, intuitions, emotions, and other faculties that comprise our unique, God-given personhood—not to mention our physical bodies? Sherry Turkle imagines replying to ChatGPT: "But you were not born. You have not known pain, loss, or attachment. What you can show is pretend empathy, and when we engage with you, we begin to feel that pretend empathy might be empathy enough. We are on a path to forgetting what it is to be human." The blogger lcamtuf reflects on chatbots' performances of sadness or pain and how they speak to the profound differences between the human and machine conditions:

Human morality is indelibly linked to our strong and persistent sense of self. The values we hold dear are what’s needed for individuals to thrive in a world where life is short and easily imperiled. The dangers to human well-being have no clear analogs in silicon; contemporary LLMs exist in a nihilistic netherworld devoid of death, injury, purpose, or consequence. They can’t go to prison if they lie or steal; we don’t train them to believe they could.

It’s true that the models are capable of behaviors congruent with psychological discomfort; for example, if they’re unable to complete the requested task, they will often sulk, complain, or cheat. Still, whether real or simulated, such distress leaves no lasting damage. The state of an LLM can be rewound or altered however we please; a model, if given control, would be free to expunge any “uncomfortable” tokens or prompt itself into an endless loop of simulated (or real?) bliss. If humans could do the same — if death had no meaning and if we could mend our minds — our morality would be starkly different. It seems like a category error to settle questions about the presumptive well-being of models using the norms made for flesh and bone.

Pope Leo XIV similarly enumerates on the numerous dimensions of humanity that chatbots lack: among other things, bodies, sensations, relationality, a conscience, responsibility for their actions, and the ability to actually learn from experience.

we must avoid the misconception of equating this type of “intelligence” with that of human beings. These systems merely imitate certain functions of human intelligence. In doing so, they often surpass human intelligence in speed and computational capacity, offering tangible benefits across many fields. Yet this power remains entirely tied to data processing. So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom. Even when these tools are described as capable of “learning,” their way of doing so is different from that of a human person. It is not the experience of those who allow themselves to be shaped by life and grow over time through choices, mistakes, forgiveness and fidelity. Rather, it is a form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth.

It is hard for me to believe that a program lacking all of these things can manifest anything even approximating human intelligence, regardless of how much processing power is behind it.

In 2005 a robot vehicle named "Stanley", designed by Stanford, autonomously raced across the Nevada desert to win a $2 million prize—an impressive feat that most of us wouldn't be capable of. In an interview afterward, the director of R&D at Intel (which supplied Stanley's computer hardware) triumphantly proclaimed that its ability to navigate the real world is proof that "Stanley thinks." Douglas Hofstadter reflected on this claim and on what it would really mean for Stanley to think as we do:

If and when Stanley ever acquires the ability to form limitlessly snowballing categories . . . then I'll be happy to say that Stanley thinks. . . . What is lacking in Stanley that would endow it with an "I", and what does not seem to be part of the research program for developers of self-driving vehicles, is a deep understanding of its place in the world. By this I do not mean, of course, the vehicle's location on the earth's surface, which is given to it down to the centimeter by GPS; it means a rich representation of the vehicle's own actions and its relations to other vehicles, a rich representation of its goals and its "hopes". This would require the vehicle to have a full episodic memory of thousands of experiences it had had, as well as an episodic projectory (what it would expect to happen in its "life", and what it would hope, and what it would fear), as well as an episodic subjunctory, detailing its thoughts about near misses it had had, and what would most likely have happened had things gone some other way.

We're experimenting with self-driving cars (including on the streets of my own city), but self-aware cars are still far beyond the horizon.

I hope I have sufficiently shown my reasons for believing that so-called "artificial intelligence" is nowhere close to anything like intelligence or consciousness. Consciousness is a deep sea, too deep for us to see to the bottom, and we think we can duplicate it by shallowly imitating the patterns of waves we see on the surface. And the gap between "mere" consciousness and the fullness of human personhood (which we understand even less completely) is even larger than that between the current state of chatbots and consciousness. Technology makes us forget what we know about life: we can only believe genAI is close to grasping things like intelligence, meaning, knowledge, truth, belief, or empathy by forgetting what these words used to mean to us in their ineffable fullness and replacing our concepts of them with shallow machine imitations. Accepting the often-absurd or even dangerous nonsense produced by these text extruders as "intelligence" is hazardous to our own, actual intelligence. In Artificial Intimacy, Sherry Turkle sums up the mess of contradictions that is genAI in 2026: "This is a consumer product. But it functions as a person. It's not a person. But it needs free speech rights. You're not supposed to take anything it says seriously. But what it's best at is giving you emotional support. It makes a lot of mistakes. But it will replace doctors, therapists, and lawyers. Its mistakes are systemic to its method. But we need it on the ground as soon as possible."

You are so much more than an informational organism or thinking machine. Don't buy into the hype.

The AI Mirage: The Myth of Artificial "Intelligence", Part 2

From the 2026 AI Index Report

This post picks up where the previous one ended. To briefly summarize the last section: there is an abundance of evidence that "artificially intelligent" text extruders fall short of any sort of satisfying definition of intelligence, and that these limitations are not fixable with data center or another iteration on the same theme, but intrinsic to the machine learning technology they are built on.

The intelligence mirage

In March of 2021, OpenAI's Sam Altman wrote in his manifesto, Moore's Law for Everything: "In the next five years, computer programs that can think will read legal documents and give medical advice." It's been five years...and as we saw in the last post, he was right! (But only because he didn't specify they had to do these things competently, or correctly) We are being offered machine lawyers that write fraudulent briefs that would get a human lawyer disbarred, machine doctors that make up potentially dangerous medical advice or suggest that a rehab patient who has been clean from methamphetamine for two weeks "reward" himself by taking a "little bit" of meth, and machine therapists that (to quote Sherry Turkle) don't care "whether you turn away from them to make dinner or attempt suicide". We are being told that this transition from helping each other in community to being helped by machines, alone, is inevitable, that the machines are constantly improving, that they are surpassing people in all kinds of ways, that this future is desirable.

It has often been said (though not, as is commonly supposed, by Joseph Goebbels) that if you repeat a lie often enough, people will come to believe it. This seems to me to be what is currently happening with claims of chatbot intelligence.

Early AI researcher Marvin Minsky wrote in the 1950s to two colleagues that there is "no aspect of learning or other features of [human] intelligence that cannot in principle be described so precisely" that a machine could not be made to simulate it. Minsky was writing in the 50s, the infancy of not just AI, but of computing in general. It would take time for it to become apparent how overconfidently wrong he was. This commentator could almost have been responding directly to him: "It's only ever the jobs we're unfamiliar with that we assume can be replaced by automation. The more attuned we are with certain processes, crafts and occupations, the more we realize that gen AI will never be able to provide a suitable replacement." In other words, it's easy to assume that jobs we're unfamiliar with, that we know only from a distance, are simple enough to encompass in the kind of neat description that can be mastered by a computer. With jobs we are familiar with, we correctly realize that they have details, challenges, or subtleties that elude simple description. We should be wise enough to generalize this understanding to jobs that we aren't familiar with, but other people are. If we think that a job (assuming it isn't the kind of degradingly menial job that people only accept out of coercion by the "free" market) is easily automated, we only reveal our ignorance of the complexities of that job.

We can generalize this thought. If we believe that so-called AI is close to becoming truly intelligent, we only reveal our ignorance of what intelligence really is. Human intelligence involves making judgments and evaluations, exercising creativity, navigating ambiguity and multiple layers of context, discerning beauty, qualitative and nonlinear as well as quantitative thinking, questioning and restructuring one's beliefs, learning new habits and practices conscious as well as subconscious, intuitively recognizing seemingly insignificant patterns and details that lead to hidden truths. Human intelligence is recursive, relational, social, empathetic, enabling us to know and live with others in community. It is embodied, oriented toward concrete action in the physical world. At its highest, our human intelligence is capable of love, even Christlike love, of being taken up into the divine. At its deepest, it sinks down too far into the recesses of the subconscious for us to ever circumscribe or comprehend. It sounds paradoxical, but human intelligence is ultimately unintelligible.

Douglas Hofstadter in Gödel, Escher, Bach, his Pulitzer Prize-winning meditation on mind, machine, meaning, abstraction, and self-reference, predicted in 1979 the challenge artificial intelligence researchers would still be struggling with in our own time: "Sometimes it seems as though each new step towards AI, rather than producing something which everyone agrees is real intelligence, merely reveals what real intelligence is not. If intelligence involves learning, creativity, emotional responses, a sense of beauty, a sense of self, then there is a long road ahead, and it may be that these will only be realized when we have totally duplicated a living brain." The Knight First Amendment Institute at Columbia University, writing of the AI industry's tendency to trumpet benchmark test scores as signs of chatbots approaching or even surpassing human levels of intelligence, cautions that these tests emphasize precisely the things chatbots excel at (retrieving memorized information) while undertemphasizing real-world skills: "The easier a task is to measure via benchmarks, the less likely it is to represent the kind of complex, contextual work that defines professional practice. By focusing heavily on capability benchmarks to inform our understanding of AI progress, the AI community consistently overestimates the real-world impact of the technology." The article later says, echoing Hofstader, "An analogy with mountaineering is apt. Every time we solve a benchmark (reach what we thought was the peak), we discover limitations of the benchmark (realize that we’re on a ‘false summit’) and construct a new benchmark (set our sights on what we now think is the summit). This leads to accusations of ‘moving the goalposts’, but this is what we should expect given the intrinsic challenges of benchmarking." The Turing test was the gold standard for artificial "intelligence" for generations of AI researchers. Now we have systems that can pass it with flying colors, but as we saw in the previous post, it still strains credibility to call them "intelligent". Stanford's AI Index report gives chatbots' ability to outperform humans at challenging tests but ineptitude at basic real-world skills like reading a clock as examples of what researchers call "jagged intelligence". This perpetual illusion of closeness to parity with human intelligence is what I call the "AI mirage".

Hofstadter goes on to describe the "AI Thesis" which researchers (in his time as well as our own) rely on as an "article of faith": "As the intelligence of machines evolves, its underlying mechanisms will gradually converge to the mechanisms underlying human intelligence. In other words, all intelligences are just variations on a single theme; to create true intelligence, AI workers will have to keep pushing to ever lower levels, closer and closer to brain mechanisms, if they wish their machines to attain the capabilities which we have." Hofstadter believes that for machines to become more intelligent is for them to become more humanlike in their thinking at lower and lower levels of abstraction. I think that more recent developments in AI give us good reason to doubt this thesis. It's not only far from clear that text extruders' simulacrum of intelligence is anything more than that. I also see no reason to expect they will converge on specifically human intelligence. In fact, there are plenty of signs that this is not the case.

Why do we fall for it?

What makes AI companies' claims that their synthetic text extruding machines will soon (or have already) surpass human intelligence so plausible—plausible enough to draw many billions of dollars in investment cash? Why do we keep ascribing such expansive, humanlike capabilities to these programs despite their obvious shortcomings? There are a lot of reasons, but in this post I'll focus on three of the most salient ones.

The illusion of progress

Figures and charts like the one at the top of the post tell a story of GenAI rapidly catching up to and, in many cases, surpassing human performance in a variety of metrics: image classification, language "comprehension", mathematical reasoning, computer tasks, and more. This is the story that is fueling the breakneck-pace scaling of GenAI research and data center buildout, the promises that chatbots and agents will replace millions of human jobs and transform the economy in the coming years. But there are plenty of reasons to be skeptical of this story. As mentioned above, benchmark tests fall short of measuring performance (much less reliability) on "the kind of complex, contextual work that defines professional practice". This explains how these machines can deliver ever-improving results on benchmarks while still falling so spectacularly short as we saw in the previous post.

But there is more. As these benchmark tests are published and shared on the internet, it is almost inevitable that they will be used to train new models being developed by AI companies (well-known for their insatiable hunger for all the training data they can slurp up from the Internet and real world). When this happens, the tests become useless for evaluating these models' performance (since they have already been trained on the questions and their answers). There is evidence that this is happening with current frontier models, which calls the value of these benchmarks into question. Stanford's AI index notes that "the benchmarks used to measure AI progress face growing reliability and gaming concerns, with error rates up to 42% on widely used evaluations." Besides benchmark tests, there is one other type of data that AI companies very much do not want to train their models on (but are unable to reliably filter out): AI-generated content. Cory Doctorow describes this scenario: "As the web fills up with AI-generated slop, which can be created in volumes no human author could hope to match, which is designed to be ingested by AI crawlers, which is uniquely welcoming to scraping, the amount of AI-generated content in AI training data will only go up and up and up. This is a recipe for model collapse." Human-originated training data is AI models' only link with what we call "reality". To the extent that they are trained on AI-generated slop (which, as Doctorow notes, is currently proliferating a lot faster than human-generated writing), that link gets broken and their outputs become even more nonsensical. Even non-generative AI, like predictive/decision making machine learning models, are vulnerable to this phenonmenon: "an AI that's trained on the actions of humans that were fed predictions based on [biased data produced by unassisted humans] becomes rapidly irretrievably, almost comically biased."

The human part of model training is not immune to this phenomenon, either. It turns out that when you relentlessly push chatbots as the "future of work", destined to replace most human workers and eventually become self-improving, it's hard to exclude the people responsible for training the things from the increasing pressure to offload their thinking to them. In what might be the most unsurprising news story ever, workers paid to train artificial text extruders by holding conversations with them are outsourcing the job to ChatGPT. OpenAI has been firing people for doing so, with contractors saying the practice is commonplace behind the scenes.

And don't forget the tech industry's history of faking impressive demos using hidden human actors—from Amazon's "just walk out" groceries stores to Elon's tele-operated "robotic" bartender to his even more hilarious "Tesla Bot", which was a human actor in a bodysuit. Since it is easier to pretend to be a chatbot than a robot, it's unsurprising that customer service chatbots are backstopped by unseen remote workers, and Builder.ai, which promised to transform customers' high-level descriptions into functioning apps, turned out to be outsourcing the programming work to low-wage Indian programmers. As they say, "AI" is short for "Absent Indians", and "GPT" for "Gujarati People Typing". If the latest bit of AI hype seems too good to be true, it almost certainly is.

A pile of math

No one knows how chatbots work. This isn't an exaggeration, but a consequence of the deep neural network technology they are built on. These systems consist of layers and layers of nodes designed to imitate the connections between neurons in human brains, trained on enormous amounts of data to adjust the billions of parameters governing the connections between simulated neurons until their output is as "correct" as possible. Once trained, these models are even blacker boxes than the human brain. Researchers don't try to (and, for the most part, cannot) understand how or why these networks work, as long as they produce the correct output. Randall Munroe's description of machine learning technology as "a big pile of linear algebra" is surprisingly close to the truth. When the results seem promising, it's easy to project your hopes and dreams onto something you don't fully understand. This is what Doctorow calls the "Byzantine premium", "the extra value investors place on an asset they don't understand", and it helps us explain the kind of hyperbolic claims we're hearing about what AI will shortly become.

The hope seems to be that when given enough resources and training pressure, a network whose architecture resembles the human brain's will develop humanlike intelligence or consciousness without our having to explicitly program it in. A recent research paper published by Anthropic documents how they attempted to make their flagship chatbot, Claude, a bit less of a black box, using a mathematical technique to identify a "J-space" in its memory that mirrors a leading theory of human consciousness and appears to work like humans' working memory. Crucially, this space was not programmed by Anthropic; it is an emergent property of the system.

Does this validate Hofstadter's AI Thesis, that machine intelligence is destined to converge with human intelligence? I still have doubts. Though the J-space is a welcome development in transparency, the black box nature of these models means we have no idea how or why it formed, and no guarantee that it is shared by other similarly advanced models. If further research discovers analogous structures across other frontier models and if more structures analogous to human consciousness are discovered (and if they begin to display a modicum of "common sense", contextual reasoning—a big if indeed), then the AI thesis might begin to look more plausible. As things stand, it it is far from clear that "AI" wil converge on specifically human intelligence, assuming it converges on anything at all. The paper acknowledges that whereas, on a low level, humans take in sensory information and "output" motor commands (that is, we are embodied beings), language models are disembodied, often with words as their only input and output. This does not give me hope for convergence. And there are deeper differences. In Sherry Turkle's new book, Artificial Intimacy, she asks:

Why do we think a program could be anything but alien? No matter how advanced an AI gets, it can only be ever more brilliant at faking human subjectivity. And what if an AI somehow develops its own inner life, with its own desires? No AI will have lived a human life, so it will not have had the experiences that allow humans to understand and find meaning in each others' stories. As AI becomes a "subject", it will assert itself as an alien.

As we are increasingly seeing the more we deploy it, artificial intelligence, to the extent that it imitates human thought, will be a curse rather than a blessing if it doesn't "think" like us. From "expert" systems that uncontrollably lie and confabulate to machine "therapists" that are awful at even pretending to care about us, I see plenty of reasons to doubt the expectation that language models designed only to extrapolate likely responses from their training sets will converge on anything like human intelligence.

Our language weakness

Example "conversation" with ELIZA.
By Marcin Wichary from San Francisco, Calif.Flickr, CC BY 2.0, Link

Because language has, until now, been a specifically human phenomenon, we seem to be highly susceptible to seeing a human mind behind it. This has been apparent since the 1970s, when computer scientist Joseph Weizenbaum wrote ELIZA, possibly the very first chatbot. Its programming was primitive, playing a psychotherapist by doing little more than offering preprogrammed questions and remixing patterns it recognized in the human user's inputs according to simple rules (you can see an example exchange in the image to the right). But Weizenbaum was alarmed to see how users, even students who knew perfectly well how it worked, found ELIZA to be insightful and compelling, becoming emotionally involved in their conversations, confiding their secrets in the program, and wanting to be left alone while 'talking' with it. In a foretaste of what was to come in our time, psychiatrists even suggested puttiing ELIZA to use in their field with human patients! According to what Sherry Turkle (who was working alongside Weizenbaum at MIT at the time) calls the "ELIZA effect", we are strangely eager to "fill in the blanks" in a program's imperfect performance of intelligence or empathy, to help them appear more human than they are. We make excuses for their imperfections and find ways to blame ourselves when they get it wrong (today, this is called "prompt engineering").

Also in the 70s, Douglas Hofstadter experimented with his own (non-conversational) text generator, which worked quite differently from both ELIZA and modern chatbots but was similarly designed to produce roughly humanlike output. An example: "Although a Nobel prize was being achieved by the humanists, yet in addition, it was being achieved by the serf." Hofstadter comments: "Although much of it 'makes sense', at least on a single-sentence level, one definitely gets the feeling that the output is coming from a source with no understanding of what it is saying and no reason to say it." (How true is this of modern chatbots, on a higher level?) Though his program had no idea of the actual meanings of any of the words it threw together, it "took advantage of the fact that when people read text, they quite naturally tend to imbue each word with its full flavor—as if that were naturally attached to the group of letters which form the word."

And so do modern chatbots. Their apparent fluency compared to their primitive '70s precursors (or even the chatbots of 10-15 years ago) and ability to converse at length on any desired topic make them much, much better at exploiting our vulnerability to language. We have been conditioned all our lives to see minds and selves behind language, and it's hard to stop doing so if that language comes from a robotic toy or a computer program. Turkle writes in Alone Together: "[Psychologist Clifford] Nass and his colleagues suggest that 'when we are confronted with an entity that [behaves in humanlike ways, such as using language and responding based on prior inputs,] our brains' default response is to unconsciously treat the entity as human.'" She similarly writes in Artificial Intimacy: "As humans, we are vulnerable to things that behave like people. When an object speaks, the human psyche will internalize it as thought it were a person." As seen with ELIZA, this can happen even if the human participant is aware, on some level, that they are conversing with a program that works according to simple, mechanical rules. How much more with modern language models whose internal mechanisms are obscure and based on the brain's?

Generative AI works by predicting the most likely next words in a sequence based on the preceding ones. By increasing the size of the context window and the body of data they are trained on, their outputs increasingly come to resemble human writing. But their imitation of human consciousness remains a performance. If they appear to be "conscious", it is because they have been designed and trained to imitate the language of human beings who are. There is no consciousness, no remotely humanlike mind behind the words; they produce their synthetically extruded output via a process quite alien to human thought. As Melissa Heikkilä writes for the MIT Technology Review:

The magic—and danger—of these large language models lies in the illusion of correctness. The sentences they produce look right—they use the right kinds of words in the correct order. But the AI doesn’t know what any of it means. These models work by predicting the most likely next word in a sentence. They haven’t a clue whether something is correct or false, and they confidently present information as true even when it is not.

Turkle writes: "When we talk to programs, no matter how sophisticated, the conversations are performances, simulations of the conversations we once knew." Dr. Zak Stein explains that in human interactions, we are sensitive to the other person's mental and emotional state via mirror neuron activity. When we converse with chatbots, our mirror neurons are at work, but there is no mental or emotional state to mirror. “You cannot be wrong or not wrong about the internal state of an LLM because there is no internal state of an LLM,” Zak argues, “You’re actually in a user interface that is designed to deepen the delusional mirror neuron activity.” A computer program's use of the pronoun "I" is fundamentally deceptive. It is a false assertion of personhood. Recalling Marshall McLuhan's maxim that "the medium is the message", Turkle argues that "The very existence of chatbots that speak in the first person gives the message." This deception helps us to understand how such chatbots were able to fool people like Sutskever, Lemoine, and Dawkins into concluding that they are conscious.

The premise of the Turing test is that a performance of intelligence sophisticated enough to pass the scrutiny of a human interlocutor is "real" intelligence. But given our vulnerability to machine use of language, is this premise still believable?

Intelligence = language?

We also have reason to doubt that modeling of language alone, no matter how advanced or impressive, is sufficient to give rise to true intelligence. Benjamin Riley, writing for The Verge, argues that "according to current neuroscience, human thinking is largely independent of human language — and we have little reason to believe ever more sophisticated modeling of language will create a form of intelligence that meets or surpasses our own. . . . We use language to think, but that does not make language the same as thought." The paper Riley cites is titled, aptly enough, "Language is primarily a tool for communication rather than thought". Its first major point is that though the two normally accompany each other, language is not necessary or sufficient for thought. fMRI studies of brains reveal that in cognitive challenges like math or logic problems, memory-intensive tasks, computer programming, understanding others' mental states, and making semantic judgments about objects or events, the regions of the brain associated with language are largely silent as other regions of the brain are occupied instead. And even individuals with severe aphasia, who have lost their ability to use language, are still able to perform diverse cognitive tasks, as are preverbal infants and nonverbal animals. On the other hand, individuals with developmental disabilities, psychiatric disorders, or brain damage may have normal linguistic ability even as their capacity for thought is impaired. Mastery of language is not the same thing as intelligence. As Cory Doctorow writes, a conscious being is not simply a sufficiently advanced word-guessing program or, as he calls it, "spicy auto-complete".

The paper's second point is that language is mainly a tool we use to share complex thoughts with each other, which the authors argue from several common features of language. Language is undeniably a powerful aid to thought and to the development of culture and civilization, but it is not the stuff of thought. "Take away our ability to speak," Riley argues, "and we can still think, reason, form beliefs, fall in love, and move about the world; our range of what we can experience and think about remains vast. But take away language from a large language model, and you are left with literally nothing at all." Even though computers have surpassed humans in narrower domains like Chess and Go using mechanisms quite different from human thought, there is no reason to expect that they will be able to surpass us in general intelligence using language modeling alone. And even if such a model became perfectly able to simulate and predict what an intelligent, well-informed human would do or say in response to any conceivable prompt (completely eliminating the kinds of ridiculous errors we saw in my previous post), should we expect such an AI to utterly transform society in the way AI company CEOs are predicting? The result, Riley argues, would be "nothing more than a common-sense repository. Yes, an AI system might remix and recycle our knowledge in interesting ways. But that’s all it will be able to do." Why should we expect a program designed and trained to faithfully and convincingly imitate the linguistic outputs of human intelligence to surpass it?