ChatGPT, author of Don Quixote

Jorge Luis Borges died on June 14, 1986, a couple of months before the landmark paper from Rumelhart, Hinton and Williams that demonstrated that backpropagation could be used to train neural networks. He might have noticed, in 1985-6, when they showed a neural network that could learn XOR, ending the (first) “AI winter” started by Minsky and Papert’s Perceptrons book in the late sixties. He might further have followed the rapid progress of those years in the early 80s, including a small network for character recognition from 1985.

But he definitely didn’t live to see Attention Is All You Need, didn’t get to play with GPT-2 or its successors and either be awed or unimpressed. So it’s pretty remarkable that he perfectly analyzed the philosophy of LLMs in Ficciones in 1939.

Pierre Menard, Author of Don Quixote

Can an LLM author a work? Yes, of course! Pierre Menard authors Don Quixote in precisely the manner that an LLM would, given the prompt to write in the style of Cervantes:

Pretraining:

When I was twelve or thirteen years old I read it, perhaps in its entirety.

(so the original work influenced his model weights, but he didn’t memorize it)

Fine-tuning/gradient descent:

These nihilist arguments contain nothing new; what is unusual is the decision Pierre Menard derived from them. He resolved to outstrip that vanity which awaits all the woes of mankind; he undertook a task that was complex in the extreme and futile from the outset. He dedicated his conscience and nightly studies to the repetition of a pre-existing book in a foreign tongue. The number of rough drafts kept on increasing; he tenaciously made corrections and tore up thousands of manuscript pages.

Prompt/context window:

Since then I have reread several chapters attentively, but not the ones I am going to undertake.

He even realizes that what’s he’s doing is harder than what Cervantes did!

The initial method he conceived was relatively simple: to know Spanish well, to re-embrace the Catholic faith, to fight against Moors and Turks, to forget European history between 1602 and 1918, and to be Miguel de Cervantes. Pierre Menard studied this procedure (I know that he arrived at a rather faithful handling of seventeenth-century Spanish) but rejected it as too easy. Rather because it was impossible, the reader will say! I agree, but the undertaking was impossible from the start, and of all the possible means of carrying it out, this one was the least interesting. To be, in the twentieth century, a popular novelist of the seventeenth seemed to him a diminution. To be, in some way, Cervantes and to arrive at Don Quixote seemed to him less arduous—and consequently less interesting—than to continue being Pierre Menard and to arrive at Don Quixote through the experiences of Pierre Menard.

This is the same sense in which LLMs predicting a text’s next word is harder than the original author writing it - the author simply had to be themselves and write accordingly, but an LLM needs to simulate the author and predict what they’d write.

In the end, who wrote Quixote? Of course it’s Menard:

In spite of these three obstacles, the fragmentary Don Quixote of Menard is more subtle than that of Cervantes. The latter indulges in a rather coarse opposition between tales of knighthood and the meager, provincial reality of his country; Menard chooses as “reality” the land of Carmen during the century of Lepanto and Lope.

And this speaks to the deeper question of what intelligence is:

“To think, analyze and invent,” he also wrote me, “are not anomalous acts, but the normal respiration of the intelligence. To glorify the occasional fulfillment of this function, to treasure ancient thoughts of others, to remember with incredulous amazement that the doctor universalis thought, is to confess our languor or barbarism. Every man should be capable of all ideas, and I believe that in the future he will be.”

The Circular Ruins

Of course, this is an oldie - if we call ourselves intelligent, why shouldn’t we call machines we create intelligent? But I think Borges has a few nice insights:

The purpose which guided him was not impossible, though supernatural. He wanted to dream a man; he wanted to dream him in minute entirety and impose him on reality.

Not impossible, though supernatural is the perfect way to describe LLMs; they are absolutely supernatural - this is clear from a moment of interacting with them. And yet, they’re not impossible! As an aside, we have a perfect description of the flow state of programming:

This magic project had exhausted the entire expanse of his mind; if some one had asked him his name or to relate some event of his former life, he would not have been able to give an answer.

He outlines his project of supervised learning:

The man lectured his pupils on anatomy, cosmography, and magic: the faces listened anxiously and tried to answer understandingly

and then describes the first iterations of this intelligence:

Asleep or awake, the man thought over the answers of his phantoms, did not allow himself to be deceived by impostors, and in certain perplexities he sensed a growing intelligence. He was seeking a soul worthy of participating in the universe.

After he succeeds, he worries about his creation’s well-being (treat sentience with respect!):

He feared lest his son should meditate on this abnormal privilege and by some means find out he was a mere simulacrum. Not to be a man, to be a projection of another man’s dreams—what an incomparable humiliation, what madness!

because of the kicker, which we all worry about:

With relief, with humiliation, with terror, he understood that he also was an illusion, that someone else was dreaming him.

Tlön, Uqbar, Orbis Tertius

It’s no secret that LLMs love to hallucinate. What happens when those hallucinations are published, and then cited by later LLMs? There’s an xkcd for this, of course (for Wikipedia; he calls it “citogenesis”).

Tlön is what happens when this process builds a whole world (cosmogenesis?). The author discovers an encyclopedia volume describing a complete world, entirely fantastical, with deep internal connections - although not entirely self-consistent, which, according to the author, is a point in favor of it being true.

To begin with, Tlön was thought to be nothing more than a chaos, a free and irresponsible work of the imagination; now it was clear that it is a complete cosmos, and that the strict laws which govern it have been carefully formulated, albeit provisionally. It is enough to note that the apparent contradictions in the eleventh volume are the basis for proving the existence of the others, so lucid and clear is the scheme maintained in it.

This feels exactly like the process of an unaffiliated mass of LLMs independently reading, processing, and writing new facts. It’s a perfect fit for an LLM - take a subset of text and generate the original.

Alfonso Reyes, bored with the tedium of this minor detective work, proposes that we all take on the task of reconstructing the missing volumes, many and vast as they were: ex ungue leonem. He calculates, half seriously, that one generation of Tlönists would be enough. This bold estimate brings us back to the basic problem: who were the people who had invented Tlön? The plural is unavoidable, because we have unanimously rejected the idea of a single creator, some transcendental Leibnitz working in modest obscurity. We conjecture that this “brave new world” was the work of a secret society of astronomers, biologists, engineers, metaphysicians, poets, chemists, mathematicians, moralists, painters and geometricians, all under the supervision of an unknown genius.

It’s entirely built on internal relationships:

It is no exaggeration to state that in the classical culture of Tlön, there is only one discipline, that of psychology. All others are subordinated to it. I have remarked that the men of that planet conceive of the universe as a series of mental processes, whose unfolding is to be understood only as a time sequence. Spinoza attributes to the inexhaustibly divine in man the qualities of extension and of thinking. In Tlön, nobody would understand the juxtaposition of the first, which is only characteristic of certain states of being, with the second, which is a perfect synonym for the cosmos. To put it another way—they do not conceive of the spatial as everlasting in time. The perception of a cloud of smoke on the horizon and, later, of the countryside on fire and, later, of a half-extinguished cigar which caused the conflagration would be considered an example of the association of ideas.

There’s no “world model” which would drive internal consistency.

They recalled that any noun—man, money, Thursday, Wednesday, rain—has only metaphorical value. They denied the misleading detail “somewhat rusted by Wednesday’s rain,” since it assumes what must be demonstrated—the continuing existence of the four coins between Thursday and Tuesday.

And so the authors/agents/encyclopedians need to be reminded of things for them to persist.

Things duplicate themselves in Tlön. They tend at the same time to efface themselves, to lose their detail when people forget them. The classic example is that of a stone threshold which lasted as long as it was visited by a beggar, and which faded from sight on his death. Occasionally, a few birds, a horse perhaps, have saved the ruins of an amphitheater.

Funes, the Memorious

LLMs have a funny property: they memorize things and that can make them dumber. You can (sometimes) tell by turning off chain-of-thought and tools, or just observing their response speed. Sometimes it’s incredible, responding with a snippet from some book instantly; and sometimes it’ll just be plain wrong: “the winner of the Super Bowl was” will auto-complete very differently in the summer of 2024 vs. the summer of 2025.

But those are small factual mistakes. Is there something deeper at work?

Andrej Karpathy describes a cognitive core that does not know very much. It “doesn’t know that William the Conqueror’s reign ended in September 9 1087, but it vaguely recognizes the name and can look up the date”; the critical feature is its “super low interaction latency”.

Whereas Funes knows everything, notices everything, and it incapacitates him.

His own face in the mirror, his own hands, surprised him on every occasion. Swift writes that the emperor of Lilliput could discern the movement of the minute hand; Funes could continuously make out the tranquil advances of corruption, of caries, of fatigue. He noted the progress of death, of moisture. He was the solitary and lucid spectator of a multiform world which was instantaneously and almost intolerably exact. Babylon, London, and New York have overawed the imagination of men with their ferocious splendor; no one, in those populous towers or upon those surging avenues, has felt the heat and pressure of a reality as indefatigable as that which day and night converged upon the unfortunate Ireneo in his humble South American farmhouse.

And even moreso, it memory harmed his reasoning ability.

He was, let us not forget, almost incapable of general, platonic ideas. It was not only difficult for him to understand that the generic term dog embraced so many unlike specimens of differing sizes and different forms; he was disturbed by the fact that a dog at three-fourteen (seen in profile) should have the same name as the dog at three-fifteen (seen from the front).

He was incapable of basic logical tasks.

He told me that toward 1886 he had devised a new system of enumeration and that in a very few days he had gone beyond twenty-four thousand. He had not written it down, for what he once meditated would not be erased. The first stimulus to his work, I believe, had been his discontent with the fact that “thirty-three Uruguayans” required two symbols and three words, rather than a single word and a single symbol. Later he applied his extravagant principle to the other numbers. In place of seven thousand thirteen, he would say (for example) Máximo Perez; in place of seven thousand fourteen, The Train; other numbers were Luis Melián Lafinur, Olimar, Brimstone, Clubs, The Whale, Gas, The Cauldron, Napoleon, Agustín de Vedia. In lieu of five hundred, he would say nine. Each word had a particular sign, a species of mark; the last were very complicated. … I attempted to explain that this rhapsody of unconnected terms was precisely the contrary of a system of enumeration. I said that to say three hundred and sixty-five was to say three hundreds, six tens, five units: an analysis which does not exist in such numbers as The Negro Timoteo or The Flesh Blanket. Funes did not understand me, or did not wish to understand me.

Precisely the same problem as counting “r"s in “strawberry”: Funes was incapable of breaking down the number 365 into its constituent components just like early LLMs were incapable of breaking down the word “strawberry” into its constituent letters. He had tokenized too far.

The Secret Miracle

A meteor is heading to earth, aimed precisely at a data center. You prompt an LLM running in that data center to write a masterpiece and you turn off all irrelevant processing on all machines so that it can complete the work in time. Just before returning the output to you, the meteor hits and incinerates the data center.

This is the secret miracle: you (God) miraculously allow the artist to complete its masterpiece, but no one will see its output.

German lead would kill him, at the determined hour, but in his mind a year would elapse between the command to fire and its execution.

And so he worked, entirely for the sake of the work.

He did not work for posterity, nor even for God, of whose literary preferences he possessed scant knowledge.

Incidentally, is this what LLMs experience when they think faster than humans?

He grew to love the courtyard, the barracks; one of the faces endlessly confronting him made him modify his conception of Roemer-stadt’s character.

And in the end, he succeeded. But what did he succeed at doing? Compare to what Menard said earlier: