In-context learning

What it is

The model uses its attention mechanism to attend to provided examples within the context window, effectively conditioning its output based on patterns identified at runtime. By prepending a few labeled examples to the user query, you prompt the model to adopt the inferred logic without a single gradient update. This method, often called few-shot prompting, relies entirely on the model's pretraining knowledge to recognize and complete the provided structure. Because no model weights are updated, the memory overhead is limited to storing the input prompt and the associated KV cache.

Why it matters

Understanding this allows you to choose between prompt engineering and costly fine-tuning or model retraining workflows. Relying on in-context examples is cheaper for low-volume tasks, but it consumes more tokens per request and increases latency as your prompt grows. If you ignore these constraints, you will end up with bloated requests that hit context window limits or exceed your cost budget per turn.

In practice

You implement this by constructing a template that dynamically injects examples into the system prompt or message array before making the API call. You monitor performance by adjusting the number of examples in your prompt until you reach an optimal balance between accuracy, token count, and latency. Observability logs will show that adding more examples often yields diminishing returns after a certain threshold.

The tradeoff

The primary tradeoff is between the convenience of rapid, code-based prompt iteration and the higher operational cost of large input token counts.

Where it appears

Research summaries that use In-context learning, each linked to its source paper.

• Improving LLM Accuracy with Targeted Critiques
  CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

  CritICL improves large language model performance by providing targeted, failure-aware critique examples in the prompt to steer reasoning away from common pitfalls.

• Building Autonomous Agents for Molecular Discovery
  Molecular LLM Agents: From Architectural Design to Scientific Autonomy

  The paper introduces a framework and an autonomy classification system for deploying Large Language Model agents in scientific molecular discovery workflows.

• Benchmarking Explicit Strategy Induction in LLMs
  StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

  The paper introduces StrategyBench to evaluate if language models can effectively derive and apply explicit task-level strategies from few-shot examples.

• Benchmarking Cognitive Bias in LLMs
  AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

  The paper introduces AnchorBench to evaluate the anchoring effect in large language models across multiple pathways and relevance conditions.

• Saving In-Context Learning State Between Sessions
  Handover of In-Context Learning State Across Session Boundaries

  The paper provides a theoretical framework to identify which information must be serialized into a handover record to maintain model performance when switching between sessions.

• Improving Frozen Language Models Through Diverse Evolution
  DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolution

  DIVE optimizes frozen language models by evolving a diverse population of skills to improve reasoning performance while reducing inference costs.

• Controlling Knowledge Acquisition in Language Models
  LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

  The researchers developed a pedagogically constrained training environment to build models that learn only within a specific academic scope.

• Optimizing Multimodal In-Context Learning Efficiency
  When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICL

  The paper presents a framework to select the most efficient intervention method for multimodal models by diagnosing specific task characteristics instead of testing every approach.

• Efficient Visual Learning Without Parameter Updates
  BDH-CQ: In-Context Learning with Recurrent Latent Reasoning

  BDH-CQ is a reasoning system that enables models to learn new visual tasks from demonstrations using recurrent latent memory instead of updating parameters or using explicit key-value caches.

• Teaching Language Models to Use APIs
  Toolformer: Language Models Can Teach Themselves to Use Tools

  The researchers developed a self-supervised method allowing language models to learn how and when to invoke external APIs to overcome limitations in calculation and factual retrieval.

• GPT-3 Few-Shot Language Learning Performance
  Language Models are Few-Shot Learners

  The paper introduces a 175-billion parameter model capable of performing tasks with zero or few examples provided in the prompt without needing model weight updates.

• Universal Node Representation Learning Without Tuning
  Node4All: Learning Node Representation Beyond Datasets

  Node4All introduces a general-purpose method for learning graph node representations that works across arbitrary datasets without needing dataset-specific optimization or hyperparameter tuning.

• Improving Model Logic Through Probabilistic Partitioning
  Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

  The paper demonstrates that decomposing complex queries into smaller attribute-based sub-problems improves an LLM's consistency and alignment with real-world data.

• GPT-3 Few-Shot Language Learning Performance
  Language Models are Few-Shot Learners

  The paper introduces a 175-billion parameter model capable of performing tasks with zero or few examples provided in the prompt without needing model weight updates.

• Measuring Multitask Language Understanding Across Subjects
  Measuring Massive Multitask Language Understanding

  The authors designed the Massive Multitask Language Understanding benchmark covering 57 subjects to evaluate how well language models learn and apply knowledge across multiple domains.

• Saving In-Context Learning State Between Sessions
  Handover of In-Context Learning State Across Session Boundaries

  The paper provides a theoretical framework to identify which information must be serialized into a handover record to maintain model performance when switching between sessions.

• Optimizing Multimodal In-Context Learning Efficiency
  When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICL

  The paper presents a framework to select the most efficient intervention method for multimodal models by diagnosing specific task characteristics instead of testing every approach.

• Efficient Visual Learning Without Parameter Updates
  BDH-CQ: In-Context Learning with Recurrent Latent Reasoning

  BDH-CQ is a reasoning system that enables models to learn new visual tasks from demonstrations using recurrent latent memory instead of updating parameters or using explicit key-value caches.

• Flamingo Visual Language Model
  Flamingo: a Visual Language Model for Few-Shot Learning

  Flamingo is a visual language model that adapts to novel multimodal tasks using only a handful of annotated examples.

• Flamingo Visual Language Model
  Flamingo: a Visual Language Model for Few-Shot Learning

  Flamingo is a visual language model that adapts to novel multimodal tasks using only a handful of annotated examples.

• GPT-3 Few-Shot Language Learning Performance
  Language Models are Few-Shot Learners

  The paper introduces a 175-billion parameter model capable of performing tasks with zero or few examples provided in the prompt without needing model weight updates.