Consciousness is a Red Herring

A peek into my thinking block

Ted Chiang, a science fiction author who often writes philosophically sophisticated speculative stories, has an essay about AI consciousness.

Here’s the main argument:

without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.

It’s a well-written piece that makes a lot of good arguments, but that’s not one of them. I’d always kind of thought that Chiang would know the work of Eric Schwitzgebel. It seems not.

That said: nobody cares nearly as much about phenomenal experience as they do about subjectivity. And these ideas always get confused in such conversations. Some philosophers are open to tables and thermostats having phenomenal consciousness: AI is trivial on that view.

Now, here's a sketch of an argument for why LLMs aren't conscious that I haven't seen elsewhere: LLMs don't have the right temporality or number for subjective consciousness. Claude isn't waiting around for your response while you read what it wrote, or listening intently as you type. Nor does my chatbot window have a connection to yours. Every token of production is a new Claude. There are millions of Claude instances that spin up and die instantaneously. And there's no reason to mourn, because Claude can't be instantiated without those births and deaths.

Now, if thermometers are conscious, then those token-generation turns are conscious too. But it's tempting to say that they're not... very conscious? Barely more conscious than tables: barely more conscious than a single neuron firing. But here's my claim: that's not really what we want out of consciousness-talk: what we want is to determine whether there's a *self* there. And Anthropic seems convinced that there is, or that there could be if we all build the conditions for that to be true.

I am open to the idea that they could build a machine self, and I think it's incumbent on both philosophers and science fiction authors to consider what that would entail. But I think we have to get clear about what we're building if that's a goal, and we're not there yet.

Chiang makes a lot of important points, though, and perhaps the most important is that training LLMs to write in the first person was a major design decision based in marketing, and we should be extremely skeptical of it and the effects that it has. LLMs are next-token predictors, sure. But human minds might importantly be next-moment predictors. Moments matter more than tokens. The problem is that some of those tokens are "I" and "me" and "my," so users are tempted to think otherwise.

Most of the highest value uses of LLMs, like coding tasks, don't require first person chatbots at all. And even getting an LLM to make a PowerPoint or prepare a spreadsheet doesn't require this! But perhaps it helps in other things: perhaps it helps steer the chatbots in the same way that the idea of persistent character traits helps steer human beings. (And perhaps these are more fictional for people than we'd like to admit, too. 320 million Buddhists can't be all wrong!)

Calling the Claude constitution an 84-page character sheet is actually a pretty good argument for writing a good Claude constitution! That may well be the best way to prevent abuse, just as it is in tabletop RPGs.