Chain of thought

What it is

When you use this technique, you instruct the model to think step by step, often by appending a phrase like 'Let us think step by step' to your system prompt. This forces the model to perform a sequential generation process where every token produced acts as an intermediate scratchpad. Because LLMs are autoregressive, these preceding reasoning tokens are included in the model's self-attention scope for subsequent tokens. This process effectively expands the computation time per query, often increasing total token consumption by 2x to 10x depending on the complexity of the task.

Why it matters

Ignoring this technique leads to lower accuracy on complex logic, math, or multi-step classification tasks where a direct answer is prone to hallucination. For engineers, it directly impacts your latency budget and token costs, as the model must generate significantly more output before completing the request. Deciding when to enforce chain of thought versus a direct response is a critical lever for balancing request cost against result quality in production features.

In practice

You implement this in code by modifying your system or user message templates to include reasoning triggers. In production monitoring, you will notice increased latency and higher token usage per request, which may require adjusting your request timeouts or budget caps for specific endpoints.

The tradeoff

The primary tradeoff is higher latency and increased cost per request in exchange for improved logical accuracy on difficult problems.

Where it appears

Research summaries that use Chain of thought, each linked to its source paper.

• Prior Scores Bias LLM Evaluation Results
  Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

  The research demonstrates that providing prior evaluation scores as metadata to LLM-as-a-judge systems causes systematic anchoring bias that distorts final judgment accuracy.

• Scaling Web Agents With Browser Sandboxes
  BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes

  Researchers built a large-scale web navigation dataset by orchestrating parallel browser sandboxes to generate diverse, high-quality interaction trajectories.

• Fixing Safety Losses During Model Reasoning
  Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

  The researchers introduce a penalty method to prevent language models from sacrificing safety protocols when they are fine-tuned for improved reasoning tasks.

• Benchmarking Explicit Strategy Induction in LLMs
  StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

  The paper introduces StrategyBench to evaluate if language models can effectively derive and apply explicit task-level strategies from few-shot examples.

• Multi-hop RAG Amplifies Transcription Errors
  Better Retrieval, Worse Robustness:How Multi-hop RAG Amplifies Upstream ASR Errors

  This research demonstrates that advanced multi-hop retrieval systems significantly increase the performance degradation caused by upstream automatic speech recognition errors compared to simpler retrieval methods.

• Efficient Reinforcement Learning for Video Models
  Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

  The researchers introduced a method called OraRL that integrates ground truth annotations as oracle rollouts to improve video model performance and reduce inference latency.

• Extracting Hidden Reasoning from Large Models
  EchoCoT: Extracting Hidden Chain-of-Thought from Large Reasoning Models

  Researchers developed a method to extract near-verbatim chain-of-thought reasoning traces from black-box reasoning models using adversarial API injection techniques.

• Efficient Navigation for Vision Language Models
  Embodied-Navigator: Point, Think, Memorize, and Align for Efficient Navigation

  TAMP-Nav improves embodied navigation by combining efficient 3D spatial grounding with selective reasoning and a multi-level reward training approach.

• Improving Long Horizon Remote Sensing Reasoning
  LongEarth-R1: Benchmarking and Aligning Vision-Language Models for Long-Horizon Earth Observation Reasoning

  LongEarth-R1 enhances long duration satellite image analysis by aligning vision language models with structured temporal reasoning and reward based feedback.

• Intelligent Routing for Image Classification Models
  MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification

  The paper introduces ARMDIL, a system that uses an MLLM router to dynamically assign images to specialized vision backbones to improve classification accuracy.

• How Instruction Tuning Affects Model Confidence
  Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity

  Researchers evaluated how instruction tuning influences the verbalized confidence and lexical diversity of rationales generated by three popular large language models.

• Improving Patent Matching with Self-Knowledge Extraction
  Self-Knowledge Retrieval Augmented Generation Framework for Patent Matching

  The paper introduces a framework that improves patent matching accuracy by using an LLM-driven process to mine technical entities and construct hierarchical ontologies for enhanced query retrieval.

• Joint Summarization and Translation for Speech
  VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation

  VoxSumm introduces a new corpus and framework for simultaneously summarizing and translating long-form spoken news content.

• Agentic Visual Search for Long Documents
  InSight-doc: Agentic Visual Perception for Long-Document Understanding

  InSight-doc uses an agentic system that zooms into document regions to reduce computational overhead and hallucination in multimodal models.

• Evaluating Vision Models on Pragmatic Incongruity
  PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models

  The paper introduces a benchmark called PragMatch to test if large vision-language models can distinguish between genuine pragmatic sarcasm and simple image-text mismatches.

• Teaching AI Models to Cite Visual Evidence
  Evidence-RL: Towards Evidence-intensive Visual Reasoning

  Evidence-RL introduces a training method that forces vision-language models to base their answers on specific image regions rather than relying on language shortcuts.

• Evaluating Theory of Mind in LLMs
  Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

  The paper introduces Avalon-ToM-Bench, a new benchmark designed to measure how well Large Language Models understand human mental states using the mechanics of the game The Resistance: Avalon.

• Finding Security Flaws Using Internal Model Signals
  Activation Probes Surface Code-Security Signals that the Model's Output Misses

  The paper demonstrates that internal residual-stream activations in LLMs contain security signals that are often lost when the model generates final text-based outputs.

• Tree Search for Language Models
  Tree of Thoughts: Deliberate Problem Solving with Large Language Models

  The paper introduces a framework called Tree of Thoughts that frames problem-solving as a search over a tree of partial solutions, allowing language models to perform deliberate exploration, lookahead, and backtracking.

• Interleaving Reasoning and Acting in LLMs
  ReAct: Synergizing Reasoning and Acting in Language Models

  The ReAct paradigm enables language models to solve complex tasks by interleaving verbal reasoning with external API actions to dynamically update plans.

• Automated Anomaly Detection in Industrial Videos
  O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning

  The O-VAD system uses object-centric tracking and chain of thought reasoning to detect and explain anomalies in industrial video sequences.

• Memory-Augmented Compression for Efficient Reasoning
  Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning

  The paper introduces a method that improves the efficiency and accuracy of chain of thought reasoning by injecting relevant, pre-computed reasoning patterns into the model prompt.

• Explainable Diagnosis for LLM Reasoning
  SymDiag: Explainable Diagnosis for LLM Reasoning via Neuro-Symbolic Verification

  SymDiag improves LLM reasoning reliability by compiling natural language chains of thought into symbolic logic for automated diagnosis and repair.

• Auditing Medical AI Reasoning Reliability
  Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought

  Researchers evaluated how medical Large Language Models reason by systematically perturbing inputs and measuring if the model's logical chain stays consistent with its final output.

• Extracting Hidden Reasoning from Large Models
  EchoCoT: Extracting Hidden Chain-of-Thought from Large Reasoning Models

  Researchers developed a method to extract near-verbatim chain-of-thought reasoning traces from black-box reasoning models using adversarial API injection techniques.

• Unlocking Enhanced Spatial Reasoning for VisionLanguage Models
  SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

  The paper introduces SCOUT, a method combining structured chain-of-thought prompting and multi-objective reinforcement learning to fix spatial reasoning bottlenecks in vision-language models.

• Streamlining Autonomous Driving Through Executable Reasoning
  XCoT-VLA: Executable Chain-of-Thought for Vision-Language-Action Driving

  XCoT-VLA replaces verbose natural-language reasoning with compact, executable tokens to improve driving performance and inference efficiency.

• Improving Language Model Reasoning with Self-Consistency
  Self-Consistency Improves Chain of Thought Reasoning in Language Models

  The paper introduces self-consistency, a decoding strategy that replaces greedy decoding with sampling and majority voting to significantly improve reasoning accuracy in language models.

• Teaching Large Language Models To Reason
  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

  The paper introduces chain-of-thought prompting, a technique where models generate intermediate reasoning steps before answering, which significantly improves their performance on complex reasoning tasks.