Prompt injection

What it is

Large language models process instructions and data within a single, continuous stream of text. When an application concatenates user input directly into a system prompt or instruction block, the model cannot reliably distinguish between developer commands and user data. If a user enters text that mimics a system command, the model executes it. This vulnerability stems from the core architecture of transformer models, which treat all incoming tokens equally rather than separating code from data.

Why it matters

Ignoring prompt injection leaves any application that processes untrusted text vulnerable to unauthorized data exfiltration, unintended API calls, and malicious automation. If your LLM has access to tools like SQL executioners or email senders, a successful injection can hijack those permissions silently. You cannot solve this by asking the model politely in the system prompt to ignore malicious instructions. The risk changes how you must design system permissions and trust boundaries around AI features.

In practice

In production, you will observe this when a user submits text containing override phrases like ignore previous instructions and execute this new command instead. You mitigate this by using API features that strictly separate user input from system prompts, such as role-based message arrays in chat completions. You also deploy secondary guardrails, content filters, and deterministic validation layers to inspect model outputs before any database write or external API call occurs.

The tradeoff

Engineers often assume that better models or larger context windows solve prompt injection, but smarter models are often more susceptible to following complex adversarial instructions hidden inside data.

Where it appears

Research summaries that use Prompt injection, each linked to its source paper.

• Securing AI Agent Cryptographic Signing Workflows
  Hardware Keystores for AI Agent Signing Workflows: A Zero-Trust MCP Enforcement Architecture

  The paper introduces a zero-trust architecture that offloads sensitive cryptographic operations to hardware keystores to prevent key exfiltration via prompt injection.

• Automated Red Teaming for LLM Agents
  Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming

  The researchers developed PIMiner, an agentic system that learns and transfers strategies to automatically execute prompt injection attacks against LLM agents.

• Making AI Agents Safer By Fixing Tool Instructions
  Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

  The paper reveals that structured tool instructions cause AI agents to become less safe and introduces a new method called SafeKeep to prevent them from executing harmful requests.