Prompt injection

What it is

Large language models process instructions and data within a single, continuous stream of text. When an application concatenates user input directly into a system prompt or instruction block, the model cannot reliably distinguish between developer commands and user data. If a user enters text that mimics a system command, the model executes it. This vulnerability stems from the core architecture of transformer models, which treat all incoming tokens equally rather than separating code from data.

Why it matters

Ignoring prompt injection leaves any application that processes untrusted text vulnerable to unauthorized data exfiltration, unintended API calls, and malicious automation. If your LLM has access to tools like SQL executioners or email senders, a successful injection can hijack those permissions silently. You cannot solve this by asking the model politely in the system prompt to ignore malicious instructions. The risk changes how you must design system permissions and trust boundaries around AI features.

In practice

In production, you will observe this when a user submits text containing override phrases like ignore previous instructions and execute this new command instead. You mitigate this by using API features that strictly separate user input from system prompts, such as role-based message arrays in chat completions. You also deploy secondary guardrails, content filters, and deterministic validation layers to inspect model outputs before any database write or external API call occurs.

The tradeoff

Engineers often assume that better models or larger context windows solve prompt injection, but smarter models are often more susceptible to following complex adversarial instructions hidden inside data.

Where it appears

Research summaries that use Prompt injection, each linked to its source paper.

• Securing AI Agents Using On-Policy Distillation
  SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

  SecOPD improves AI agent security against adaptive prompt injection by using on-policy distillation to provide fine-grained training signals that distinguish between trusted instructions and malicious data.

• LLM Verification Layers for Robot Autonomy
  Agentic Harnesses: LLM-Driven Verification Layers for Robot Autonomy

  The paper introduces a multi-layered verification framework that uses an ensemble of LLM judges to validate robot action plans for safety, security, and ethical alignment before execution.

• Exposing Hidden Reasoning Traces in LLMs
  Stealing Reasoning Traces from Proprietary LLM APIs

  Researchers discovered an architectural flaw in how major LLM providers handle encrypted reasoning traces, allowing them to decrypt and expose proprietary data.

• Securing Large Language Models with Semantic Overlays
  Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

  Semantic Overlays add an out-of-band annotation layer to input streams to help models distinguish between trusted instructions and untrusted content, significantly reducing prompt injection success rates.

• Assessing Indirect Prompt Injection in DeepSeek Harness
  Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection

  The paper uses an automated framework to evaluate how untrusted external data can manipulate agents in the DeepSeek Harness framework into performing unintended actions.

• Testing Agent Memory Against Poisoning Attacks
  Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Ranking

  The paper demonstrates that existing content screening and provenance ranking methods fail to reliably defend agent memory systems from adversarial data injection.

• Assessing Indirect Prompt Injection in DeepSeek Harness
  Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection

  The paper uses an automated framework to evaluate how untrusted external data can manipulate agents in the DeepSeek Harness framework into performing unintended actions.

• Automated Security Testing for LLM Agents
  ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

  ToolHazard provides a scalable framework to automatically synthesize stateful environments and generate adversarial tasks for evaluating and aligning LLM-based agents.