When an AI is running, it is so effortless that we feel it cannot possibly be doing anything intelligent. Is this fair?

I read some great comments on Mastodon: “#chatgpt is like using a car in a footrace; you’re covering distance but without the soul, sweat, or heart”, and further, “sometimes you really need to get to the post office before it closes, and sometimes you want to feel the wind in your hair.” It’s true! There are situations where a lack of effort seems like cheating, or devalues the accomplishment.

But other cases are different. For example, there is the story of the old mechanic (one version, more versions), which goes like this:

A customer brings their car to the old mechanic. He listens to the engine for a minute, then taps it with a hammer, and the problem is fixed.The bill is $100. The customer is outraged and demands an itemized bill.

  • Tapping the engine: $5
  • Knowing where: $95.

Now the real point is that knowing where to tap isn’t as easy as it looks: you have to put in a lot of effort to make it look, eventually, easy. The story works because it plays with our concept of effort and reward. The mechanic’s real effort, which deserves financial reward, is displaced: it is in his years of training and experience.

But let’s think about some other things that are just as hard for humans, but in a different way: wicked problems. A wicked problem “is a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize”.

Wicked problems arise all the time in the real world, for example in social policy, and in engineering, design, and art. The setting I’m most familiar with is in architecture:

Schön characterises designing as a reflective conversation with the materials of a situation. The designer is making a move in the design process that causes changes which, in turn, talk back and provoke a response by the designer to the situation’s back-talk. – Ammon.

The opposite of a wicked problem is a tame problem – the type of problem that arises in chess, and often, perhaps, in science. In a tame problem, there’s a definite end-point and you can’t jump out of the system. Some humans have good intuition for approaching wicked problems, and some humans are good at chess, but they seem quite different skills. (They are also sophisticated, adult skills, so this isn’t an example of Moravec’s paradox).

Now, a crucial aspect of approaching a wicked problem is that the problem is not, initially, well-defined. It’s not even formalised. You have to really work with the problem to understand it. You can try to formulate a solution, but you know all along that it’s not going to be the solution – you’re building it to throw it away – but it still has value, because even if imperfect, a solution helps to understand the problem a bit more. Working with wicked problems is always iterative in this way. It’s effortful.

AI systems, so far, do nothing like what humans do with wicked problems. An AI system seems effortless. To be specific, neural networks are fast and simple at runtime. A language model is “just” predicting the next word, “merely” multiplying matrices. It seems impossible to ascribe any intelligence to something so simple.

But in fact, like the old mechanic, AI systems displace the effort. A language model might require months of training and many state-of-the-art GPUs. It might require terabytes of training data. Most important, it requires a formalisation of the problem, by a human. Maybe there is some blood, sweat and tears in the formalisation, in the training data, and in the training? Should we give the AI some credit for this – say $100 – even if only $5 of that is for what it does at runtime?

The main feature of ChatGPT that I believe causes people to take it a bit more seriously as an AI is that in the chat interface we see the illusion of effort. We see the AI gradually reframing the problem and gradually improving the answer. We don’t get an instantaneous, take-it-or-leave it answer, which would tend to leave us cold. Not only does the AI “know”, but by referring back and iterating it shows that it “knows that it knows”. (Scare quotes still needed, though.)

But all this is with a human in the loop. Humans do that too (ie a human struggles with a problem, with another human in the loop). But we also do it alone: when we turn a problem over in our mind, we play the role of the human in our own loop. When thinking deeply, effortfully, we spin up an internal model to interact with. (There are interesting speculative connections to self-talk and origins of language and consciousness, here.) I think something like that is in the near future for AI too.