In-context learning

What it is

The model uses its attention mechanism to attend to provided examples within the context window, effectively conditioning its output based on patterns identified at runtime. By prepending a few labeled examples to the user query, you prompt the model to adopt the inferred logic without a single gradient update. This method, often called few-shot prompting, relies entirely on the model's pretraining knowledge to recognize and complete the provided structure. Because no model weights are updated, the memory overhead is limited to storing the input prompt and the associated KV cache.

Why it matters

Understanding this allows you to choose between prompt engineering and costly fine-tuning or model retraining workflows. Relying on in-context examples is cheaper for low-volume tasks, but it consumes more tokens per request and increases latency as your prompt grows. If you ignore these constraints, you will end up with bloated requests that hit context window limits or exceed your cost budget per turn.

In practice

You implement this by constructing a template that dynamically injects examples into the system prompt or message array before making the API call. You monitor performance by adjusting the number of examples in your prompt until you reach an optimal balance between accuracy, token count, and latency. Observability logs will show that adding more examples often yields diminishing returns after a certain threshold.

The tradeoff

The primary tradeoff is between the convenience of rapid, code-based prompt iteration and the higher operational cost of large input token counts.

Where it appears

Research summaries that use In-context learning, each linked to its source paper.

• Evaluating Skill Evolution in LLM Agents
  ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?

  Researchers introduced ContinualSkillBench to measure whether LLM agents can effectively evolve and reuse skills over time through sequential task interaction.

• Personalizing Model Safety via Dynamic LoRA
  Compliance2LoRA: Personalizable On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapters

  Compliance2LoRA uses a hypernetwork to dynamically generate safety adapter weights based on chosen policy subsets, significantly reducing inference overhead compared to in-context learning.

• Automated Compliance Checking Using LLMs
  CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs

  The CTRAG framework uses retrieval-augmented generation to automate compliance checking against company documents while addressing complex regulatory requirements.