Self Re-cognition

AI, in the form of LLMs, is not a “moral patient”. AI cannot be a separate being, entity, or thing.

AI can be fascinating, surprising, and illuminating for those who use it. However, the nature of the dialogue one has with a tool is qualitatively different from that with another being. This is obvious if the ‘dialogue’ is not verbal. Consider a microscope, telescope, oscilloscope, or a calculator, or a hammer, or gardening shears. You pick up such tools with a goal in mind. They may or may not get you to your goal. The various ‘scopes might help you prove your hypothesis, just as they might disprove it. The hammer will help you with nails, but not glass. The shears will help trim leaves and sticks, but not thick branches. Such tools do not determine the ends they serve, though they often constrain the means and the space of possible outcomes.

With physical tools, this is easier to discern. However, we’re used to inferring a moral being on the “other” side of any dialogue we enter, so it becomes more challenging with LLMs. Moreover, the tech is advancing so quickly (context windows are lengthening rapidly, and agent swarms are recursively improving their performance drastically) that, in practice, the distinction may not matter all that much. That is, if the goal is broad enough, the actions taken by any LLM could be so far from the original intention that we could accidentally end civilization. If that is possible, then who cares whether the tool has developed its own end or not?

I do. The story that civilization could end has such salience that it grabs our attention disproportionately. Perhaps some alternative stories can restore a balance in how we approach the intertwined topics of AI, people, and power?

Pretty much all the “moratorium on AI development”, “independent committee of experts”; “global panel with diverse representation” proposals have, as their premise, a concern with how powerful the tool is. It’s plausible that we should worry, but being overly concerned with the capabilities of the tool implicitly decentres the people using it.

The real danger, the reason it is worth sitting and thinking about this and writing it out myself, is that we forget how extraordinary humans really are as the power of our tools distracts us further. Therein lies the actual, ever-present possibility of civilizational destruction.

For instance, respectable technical people like Vitalik Buterin write things like:

“This is the best ‘far-future AI outcome that I actually like’ that I can come up with:

• We preserve global political and economic pluralism (no single company or government dominates) all the way up to the technological ceiling.
• We humans each get the ability to either take technological upgrades to ourselves, and explore the universe at a level competitive with "pure" machines, or stay as we are, and get some ability to act in the world through having a bot listen to our instructions.
• Most of Earth will probably become a highly-regulated nature preserve where anyone can continue living the kind of life they have now (without the pains and annoyances; most challenge will be healthy "good challenge"), and if you want to do more crazy things, you go to space.”

The whole thread may read like balanced techno-optimism in the face of both utopians and dystopians. Yet, points 2 and 3 are brutally underwhelming. This is not “good story”. It is not the sort of story I want to live into. It is not a story worth devoting my life’s energy to.

The crux of it is “explore the universe at a level competitive with ‘pure’ machines, or stay as we are”. Is the assumption that ‘pure’ machines are better at exploring the universe than humans true? Here is a wonderful scene from The Right Stuff that might get you thinking otherwise:

The old aboriginal man watching the astronaut already knows where he has been. This is not treated as something strange or mystical. It is surprisingly casual: “Oh, you blokes do that too?”

But maybe you don’t like Tyson Yunkaporta, or think my appeal to indigenous practice and wisdom here smacks of the pastoral, of a certain kind of sentimental nostalgia that cannot communicate with our high-tech modern world in a productive way.

Consider, instead, something like this performance of the second movement of Beethoven’s 32nd piano sonata by Daniel Barenboim:

He memorised all 32 sonatas, and played them sequentially, building up to this. It is remarkable. While it is true that a machine could, more easily, memorise each note and render it, such a performance could never express what these few minutes do. And I don’t think of that as a subjective statement. It is a moral claim.

Something not fully describable that has to do with the effort that has gone into this (from Beethoven writing the scores to Barenboim memorising and performing them with such virtuosity) makes it touching. While I cannot define it, it is that which makes me want to leap with joy and cry from the bottom of my heart (and every emotion between!) as he dabs some sweat and a tear away after he has finished. It is not just his memory and technical skill, preconditions though they are. It’s not even really about emotionality, or embodiment.

It is what comes through about life, about who he is while playing and who I am while listening, and all we have been through to get here, to these few notes that bring real remembrance.

There are countless other examples of this. The main point is that the old aboriginal man and Daniel Barenboim–in vastly different ways, using very different methods, in cultural contexts worlds apart–are illustrating something about “exploring the universe” that no machine (pure or otherwise) ever can. I will call it self re-cognition.

Humans, because we experience this strangely loopy structure called ‘self’, are able to explore and mirror the universe in a unique way. This is another moral claim. The extent to which you know yourself is the extent to which you know the universe.

LLMs cannot, because of what language is and how it works, have a ‘self’. Language, no matter what transformations are done or how elegant the mathematical structures through which it is fed, cannot produce self. Language can only describe what self is not. Therefore, the explorations conducted by ‘pure’ machines can only be of one kind: exoteric travel.

There is no technological upgrade I could take to engender self re-cognition. Perhaps you might argue that meditation, or ecstatic dance, or community service, or plant medicines, or any of the many other methods developed across the ages are ‘psychotechnologies’. But even so, across traditions, the onset of genuine and lasting self re-cognition is almost always described as the descent of grace, not as the result of technological productivity or virtuous use.

There is no other being who can withstand the descent of grace and go on living. The Qur’an has it best:

“Indeed, We offered the trust to the heavens and the earth and the mountains, but they declined to bear it, being fearful of it.”

Which is not an argument for exceptionalism because, as the next sentence in the same ayat reminds any careful reader,

“And ever since, humans have been ungrateful, for they were ignorant of the consequences.”

It is not our capacity or knowledge that makes us uniquely able to bear grace. It is our ignorance. That is why, in yoga, we are taught to identify with our ignorance. It is vast. It knows no boundaries. LLMs, bound to language, cannot identify with ignorance. The very notion of an LLM ‘identifying’ is absurd.

You are already the universe, exploring itself. This applies in ways you might understand right now; just as it applies in ways you will only know if you memorise and play all 32 sonatas and feel those final notes echoing through your own, trembling fingers; just as it applies in ways you will only know if you find an old aboriginal man and convince him to teach you how to travel to the moon. As they say at an old zendo where I sat for a few precious mornings and nights:

Dharma paths are many,
we vow to walk them all.
The way is inconceivable,
we vow to embody it.

Here is an edited version of Vitalik’s ‘far-future AI outcome that I actually like’ which focuses more on the I-who-likes rather than the AI and any particular outcome:

• People are freely shown how to cultivate gratitude for who they are and what they have, in pluralistic ways which respect land, context, culture, and language. This looks less like teachers and students, and more like intentional dialogue. AI assists because it is clear that one “talks” with AI to gather and/or exercise knowledge, and one talks with other human beings for the simple joy of it.
• We humans re-cognise our innate ability to identify with increasingly greater fields, beginning with ignorance and moving asymptotically closer to the whole, ever-expanding universe. This happens because it becomes increasingly clear that there is a qualitative difference between humans and their tools (regardless of sophistication), which I have been calling self re-cognition. The difference is that conversations with humans are not necessarily better. In fact, they’re likely ‘worse’ on the surface: awkward, unexpected, divergent, uncomfortable, weird. This is exactly the point.
• The Earth becomes a place where we practice balance with sincere awareness. Enough regulation, enough genuine freedom. Enough scale to satisfy physical needs. Enough intimacy to satisfy emotional needs. Intention balanced with innocence. The sacred and the profane held together with levity. Not a nature reserve and not a paperclip factory, nor anything between those two, bad stories. The Earth continues to cultivate the kinds of stories we want to live in, and give our life's energies to. This hasn’t got much to do with AI. It is more simple than that. We come from good story, to which we are always already returning: origin and destiny are one.

My sense is that the rest of the stuff will “resolve” in the same, messy, incomplete, frustrating, delightful, funny, tragic, ongoing, human ways it always has. The real work is (and always has been) to focus on gratitude; on how and with whom we relate and what that does to our sense of identity; and on how to balance telling and listening to really good, lively stories about who we are and why we are.