Ai Reasoning Planning Vs Predicting

Why AI Needs Planning, Not Just Token Prediction | AI21

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Co-authored by Barak Lenz, Yonatan Belinkov, Subbarao Kambhampati, Kevin Leyton-Brown, Yoav Shoham .

There is great interest today in AI that can reason better than traditional LLMs (if the seven-year practice of building LLMs can be called a tradition). “Reasoning” is of course a loaded term, but in the context of today’s AI it has come to mean exploring chains of tokens (including internal “thought” tokens, hence the term “Chain of Thought” or CoT), rather than myopically predicting only the very next token. Indeed, these CoT models are sometimes called Large Reasoning Models, or LRMs. In their incarnation thus far – for example, in the recent LRMs of OpenAI, Google, DeepSeek, X.AI, Anthropic, Alibaba, and others – a base LLM capable of producing solution guesses preceded by intermediate “thinking” tokens, is post-trained with a Reinforcement Learning (RL) phase to optimize for producing chains ending in correct answers. Then, at inference time, that model is allowed to pursue such chains up to a certain budget. The result has demonstrably improved performance on multiple tasks.

The point of this conceptual piece is to explain why this is a suboptimal way of exploring the space of possible answers to a given input, and describe a planning-based alternative. The purpose here is not to describe in detail the AI system built at AI21 – that’s a topic for a different paper, and we’ll only touch on it here – but to explain in more general terms how we see this space.

At a high level, here are the main messages:

Think actions, not tokens. An action is any specific invocation of a given tool-–a specific prompt to an LLM, a specific web search, a specific database call, etc. Look for the best action sequences to take, not the best tokens to predict.

The space of action sequences is huge. Explore it via explicit decision-theoretic planning, taking into account distributions over both answer quality and cost (in terms of time, compute and money). The plan is the outer loop; LLMs (and other tools) the inner.

At training time, teach the system how to plan. At test time, create a specific plan and execute it, interleaving planning and execution.

Involve the user throughout the process, especially at test time. The user is both the boss of the system and a resource for providing it with real-time corrective information.

Current “Reasoning” Models

Let’s begin by looking at some inherent limitations of the current LRM approach, and then discuss the alternative, as we see it. We should caveat this by saying that most LRM developers haven’t provided much detail about their models, so this is based on the one model – DeepSeek’s R1 – that was described in a fair amount of detail, and informed guesses about the rest.

We see two main categories of algorithmic limitations of LRMs. The first is inefficiency. At training time, for an input to carry a signal for the RL, the output must culminate in success sufficiently often. Even in the case of a not-COT LLM, since the operation of the LLM is stochastic, it is common to sample the output to each input K times, with the hope that at least one is successful. K is constant across all inputs, which is inefficient: for easy cases it’s an overkill, for hard cases it’s not enough, and for very hard cases no amount of sampling would be enough and so it’s a waste even to try. With CoT the problem is even worse, because CoT blows up the search space with “thinking tokens”. This means that for any given sampling budget—measured in tokens—LRMs explore an exponentially smaller subset of the tasks than LLMs. If LRMs exponentially increased the probability of finding a good answer within a reasonable budget, that would offset the downside of the larger search space. Yet that doesn’t seem to be the case empirically, and LRMs tend to run for a long time at great expense.

Another efficiency challenge is that the reward signal is given only at the end of the chain, and thus can’t be intelligently apportioned among the tokens in the chain. (This last point isn’t inherent to RL, and reflects the way it is used – again, as far as we can tell – in current LRMs. But it’s possible to apply Process Reward Modeling to provide explicit rewards to intermediate steps in the chain). In addition, you have space complexity limitations, since all the training is done by feeding an increasing set of chains into the context window. All these inefficiencies are amplified manyfold as you increase the set of tools, growing the search space correspondingly. Similarly, at test time there is inefficiency. An LRM is at the end of the day a language model, and at test time what you can do with it is feed it the input and do a forward pass. You don’t have much control over the length of the chains, can’t parallelize, can’t adapt it to the specific state of the compute resources, et cetera. Some of these limitations can be mitigated to some degree. It’s relatively easy to bias the model to produce longer or shorter chains; it’s possible to change the implementation so that chains can be followed in parallel and the results aggregated through some simple mechanism such as majority voting; et cetera. But we find these heuristic methods relatively weak compared to explicit control via deliberate planning, as we discuss below.

A separate, and perhaps deeper, algorithmic limitation of LRMs is their inability to generalize in a robust way. LRMs don’t really “reason” in the usual sense of the term. Reasoning is usually understood to be the rigorous application of trusted procedures, with predictable and correct outcomes, which apply to a wide class of tasks, e.g., the procedure for multiplication which applies to all valid inputs. LRMs don’t do that; instead, they probabilistically pursue useful chains of tokens. Sometimes those chains correspond to what one might call reasoning, but in general they do not, leading some of the authors of this document to describe these CoT models – only somewhat tongue in cheek – as “chain of thoughtlessness”, “large musing models” and “large mumbling models”. Whatever you call them, these models are best thought of as wonderful free associators rather than reasoners per se. There is ample evidence that these are not theoretical concerns but real shortcomings. For example, a recent paper shows that...