Tree of thoughts
A prompting framework that forces the model to generate multiple alternative reasoning branches and evaluate them to select the most promising path toward a final answer.
What it is
Instead of generating one linear response, the system prompts the model to produce multiple potential next steps or hypotheses. A secondary call or a scoring function evaluates these branches for quality or logical coherence. You then prune the low-scoring branches and continue expanding only the viable ones. This process usually involves 3 to 10 parallel branches per step and can increase the total token count by an order of magnitude.
Why it matters
Standard LLM inference produces a single stream of tokens, which often leads to local optima or early errors in complex logic. Implementing this approach allows you to solve multi-step problems that are otherwise prone to hallucination or logical dead ends. You must care about this because it drastically changes your latency profile and cost per request by multiplying the number of tokens processed for a single user query.
In practice
You typically implement this as an orchestration layer using a loop that calls your LLM API to generate candidates and an evaluation function to rank them. You will observe a direct trade-off between the depth of the search tree and the time-to-first-token in your monitoring tools. It is effectively a search strategy over the model's output space rather than a built-in API parameter.
The tradeoff
You trade significantly higher inference costs and request latency for increased accuracy in complex reasoning tasks.