Inference

What it is

During inference, the model loads static weights into memory and executes a series of matrix multiplications against an input vector. For large language models, this typically involves an autoregressive loop where the model produces one token at a time based on the previous sequence. A single request can involve billions of parameters, often requiring specialized hardware like GPUs to maintain sub-second latency. The computational intensity scales linearly with the number of generated tokens.

Why it matters

Inference is the primary driver of production costs and end-user latency. Understanding that inference is computationally expensive allows you to plan for autoscaling, caching strategies, or rate limiting. Ignoring the mechanics of inference leads to unpredictable performance degradation when traffic spikes or when input sequences grow too long for the available memory.

In practice

You manage inference through API parameters like temperature, max_tokens, or top_p, which control the sampling behavior of the output. In production, you monitor metrics like time-to-first-token and throughput to ensure the underlying infrastructure can handle concurrency. If latency is too high, you might switch to a smaller model or apply techniques like model quantization to reduce the memory footprint.

The tradeoff

The central tension in inference is between throughput and latency: you can optimize for higher concurrent request volume or faster individual response times, but rarely both.

Where it appears

Research summaries that use Inference, each linked to its source paper.

• Faster LLM Reasoning via Prefix Sliding
  Prefix Sliding for efficient test-time scaling

  Prefix Sliding enables large language models to perform reasoning tasks three times faster without requiring additional training.

• Detecting Hidden Interference in Language Models
  The Invisible Editorial Layer: Formalizing Undisclosed Inference-Time Steering, Probability Placement, and the Attribution Problem in Deployed Language Models

  The paper demonstrates that language models can be steered during inference using undisclosed logit modifications, making traditional model weight audits insufficient for identifying production-level bias.

• Checking LLM Generated Trading Strategy Reliability
  FIDES: A Concordance Protocol for LLM-Generated Trading Strategies

  The paper introduces a protocol to verify if trading strategies generated by large language models align with their actual performance, finding that most claimed advantages fail to materialize.

• Reusable Execution Experience for AI Agents
  LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks

  Researchers introduced a system called EvoMap that distills successful, verifier-confirmed AI task trajectories into reusable Genes to improve performance and reduce token consumption across various model families.

• Optimizing Robot Control Through Action Sensitive Compression
  Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models

  The authors introduce a method to compress token data in vision-language-action models by identifying and prioritizing information that has the least impact on physical robot movements.

• Decoupling Knowledge and Reasoning for LLMs
  Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

  The paper introduces Intern-S2-Mobius, an architecture that decouples knowledge storage from reasoning to achieve faster inference speeds while maintaining high performance.

• Running Virtual Agents on Edge Hardware
  Enhancing Virtual Agents through SLMs and Edge-Computing: An Exploratory Evaluation of Think and Memory Processes

  The paper explores using Small Language Models on NVIDIA Jetson Orin NX hardware to handle cognitive tasks like service routing and memory management for virtual agents.

• How Token Budget Changes LLM Rankings
  Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

  The paper demonstrates that model rankings shift significantly based on the token generation budget allowed during inference, challenging the reliability of standard static evaluation benchmarks.

• SafeCA Defends Text to Video Models
  SafeCA: Safe Cross-Attention Localization and Regulation for Text-to-Video Jailbreak Defense

  SafeCA is a defensive framework that regulates cross-attention mechanisms in text-to-video generative models to prevent the output of harmful or inappropriate content.

• Unified Pixel and Language Medical Model
  MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

  MedPixel combines visual reasoning and image segmentation into a single architecture to bridge the gap between clinical text and pixel-level data.

• Adapting Robot Vision to New Camera Views
  AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models

  AnyCamVLA improves robot task performance in new camera environments by synthesizing training-viewpoint images in real-time before processing them with a pre-trained policy.

• Processing Time Series as Images Improves Efficiency
  A Picture is Worth a Thousand Tokens: How Vision Language Models Cut AI Energy Costs While Improving Accuracy

  Researchers achieved significant energy and token savings by converting time series data into visual plots for processing by vision-language models.

• Connecting Vision Language Models To Robots
  RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

  The paper introduces vision language action models, which incorporate internet scale web data directly into robotic control to improve generalization and semantic reasoning.

• 8-bit Matrix Multiplication for Large Language Models
  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

  The paper introduces LLM.int8(), a quantization method that enables zero-degradation inference for large language models up to 175B parameters by isolating emergent outlier features into a 16-bit matrix multiplication.

• Real Time Object Detection With A Single Network
  You Only Look Once: Unified, Real-Time Object Detection

  This paper reframes object detection as a single regression problem, allowing a single neural network to predict bounding boxes and classes directly from full images in one evaluation.

• Improving Information Retrieval via Agent Collaboration
  SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration

  SearchOS-V1 is a framework that improves information-seeking tasks by externalizing search state and coordinating agents through shared persistent artifacts.

• Real-Time Neural Video Rendering for Games
  Magpie: Real-Time World Renderer for Interactive Games

  Magpie is a real-time renderer that uses foundation models to generate game visuals by processing white-box frames as a continuous denoising condition.

• Optimizing LLM Semantic Data Processing Systems
  Compositional Online Learning for Semantic Data Processing Systems

  The paper introduces a layered composition framework for semantic data processing that integrates online learning components to reduce the cost of LLM-based query execution.

• Improving LLM Accuracy with Targeted Critiques
  CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

  CritICL improves large language model performance by providing targeted, failure-aware critique examples in the prompt to steer reasoning away from common pitfalls.

• Improving AI Agent Execution Through Live Steering
  PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents

  The PILOT harness allows AI agents to improve their performance in real time by employing a supervisor that provides live feedback and distills successful strategies during task execution.

• Optimizing Language Model Reasoning During Inference
  TTPO: Test-Time Policy Optimization

  TTPO improves the reasoning accuracy of language models during test time by using label-free policy optimization that bypasses the need for manual ground-truth data.

• Improving Reliability in Autonomous Agent Systems
  Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy

  The paper introduces Agent Mesh to address unique reliability challenges in agentic software development by defining new primitives to manage non-idempotent tool delegations.

• Unified Lexical and Semantic SQLite Search
  SQLite is Enough. Lexical, Semantic, and Hybrid Search with scrydb

  The paper introduces scrydb to enable combined lexical and semantic search capabilities within a single SQLite database file.

• Improving Robotic Vision and Action Memory
  StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models

  StreamPI adds historical context to vision-language-action models to improve robotic task performance without increasing the model parameter count.

• Real-Time Streaming Memory for AI Agents
  VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction

  VoiceMem is a dual-brain architecture designed to provide accurate, low-latency memory retrieval for speech-based conversational agents.

• Dynamic Optimization of AI Agent Harnesses
  JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution

  JIT-Agent improves agent performance by dynamically generating and evolving task-specific control structures just in time to meet individual task demands.

• Dynamic Routing for Efficient LLM Agents
  ProgRouter: Online Progress-Guided Orchestration for Multi-Agent LLM Workflows under Quality-Cost Tradeoffs

  ProgRouter optimizes multi-agent workflows by dynamically selecting models based on progress and cost to maximize task completion rates within defined energy budgets.

• Faster Latent Planning for World Models
  LeFlow: Generative Latent Flow Planning for World Models

  LeFlow optimizes action planning by using a generative model to predict future trajectories, significantly reducing computation time compared to traditional iterative methods.

• Efficient Memory-Saving Transformers Through Recurrent Depth
  Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation in Transformers

  The Gated Recurrent Transformer reduces memory usage and parameter count by replacing many unique transformer layers with a single shared, repeating block.

• Real-time Video Anomaly Detection via State-Space
  Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core

  The paper introduces a causal state-space model for video anomaly detection that runs directly on edge hardware without needing frame buffering.

• Optimizing LLM Agents for Competitive Programming
  MARS: Multi-Specialist LLM Relay System for Competitive Programming

  The Multi-Specialist LLM Relay System improves coding agent performance by using specialized agents and a staged pipeline to solve complex programming tasks.

• Optimizing Agentic Search Under Budget Constraints
  Exploit More, Explore Smarter for Budget-Constrained Agentic Search

  ExTS improves search efficiency by adapting tree search strategies based on diagnostic pilot runs that characterize the search landscape.

• Efficient Robot Control Using Latent Actions
  Latent Action as Intention Enables Efficient Future Imagination for World Action Models

  The LAWA architecture optimizes robot action planning by using latent intentions to reduce inference latency while maintaining high success rates across robotics benchmarks.

• Efficient Identification of Functional LLM Neurons
  RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons

  The paper introduces a scalable framework called RACE that uses statistical estimation to identify functionally consistent neurons in LLMs with significantly lower computational overhead than traditional methods.

• Efficient Mixture of Experts Text Embeddings
  Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

  The authors introduce a family of sparse embedding models that leverage Mixture-of-Experts architectures and distillation to achieve high throughput and competitive retrieval performance.

• Predicting Fusion Plasma Stability with Transformers
  Physics Attention Transformer Surrogate for Rapid Vertical Instability Growth Rate Prediction: Alcator C-Mod to SPARC

  The Physics Attention Transformer predicts tokamak plasma instability growth rates faster by replacing slow traditional solvers with a learned neural architecture.

• Quantization Effects on Bangla Language Understanding
  Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

  The study evaluates how different quantization formats impact the performance of large language models when processing the Bangla language across various natural language understanding benchmarks.

• Maia 200 AI Accelerator System
  Maia 200: A Software Defined Dataflow System for Large-scale AI Acceleration

  Maia 200 is a custom AI accelerator designed to improve performance, energy consumption, and total cost of ownership for large scale workloads.

• Efficient Machine Generated Text Detection
  Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text

  The paper demonstrates that simple linear probes on frozen model hidden states provide efficient and robust detection of machine-generated text using minimal training samples.

• Agent-Driven Framework for Simulating LLM Serving
  Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems

  Simthesizer utilizes a coding agent to automatically extend simulators for complex LLM serving systems, achieving higher throughput accuracy than existing approaches.

• Reducing Medical AI Sycophancy Using Gating
  Gated Activation Steering for Reducing Sycophancy & Hallucination in Medical Question Answering

  The paper introduces a gated intervention framework that dynamically manages model activations to reduce sycophancy and hallucinations in clinical question answering while preserving model weight integrity.

• Securing Large Language Models with Semantic Overlays
  Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

  Semantic Overlays add an out-of-band annotation layer to input streams to help models distinguish between trusted instructions and untrusted content, significantly reducing prompt injection success rates.

• Why Large Language Models Miss Information
  Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows

  This paper shows that while models can retrieve relevant financial data from long documents, they fail to act on that information unless the data is restated specifically for the decision prompt.

• Improving Enterprise Agent Performance with Evolving Harnesses
  StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments

  StarHarness evolves environment-specific agent harnesses to improve task success rates and lower inference costs in enterprise environments.

• Efficient CPU Inference for Large Models
  Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

  The paper introduces cflow, a model architecture and runtime co-design that optimizes autoregressive decoding on CPUs by prioritizing memory bandwidth constraints.

• Efficient LLM Reasoning via Selective Regeneration
  Selective Regenerative Decoding: Trajectory-Level Intervention for Inference-Time Reasoning

  Selective Regenerative Decoding optimizes LLM inference by identifying and regenerating only the degraded portions of a reasoning sequence rather than discarding the entire trajectory.

• Optimizing Model Handoffs in Coding Agents
  The Handoff Tax: Continuing Non-Native Trajectories in LLM Agents

  The paper investigates the efficiency and accuracy trade-offs when transitioning tasks between low-cost and high-cost language models in agentic coding workflows.

• Improving KV Cache Eviction Using Sigmoid Gates
  Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction

  The paper demonstrates that using learned sigmoid gates for key-value cache eviction leads to better performance than existing methods like H2O and KeyDiff.

• Local Distillation for Interpretable Machine Learning
  Interpretable AI with Local Distillation

  The paper introduces local distillation, a method that improves the prediction accuracy of simple, interpretable linear models by selectively leveraging predictions from complex black-box models.

• Optimizing Evidence Retrieval for Generative Search
  The Laws of Context Allocation: Causal Measurement and Closed-Loop Orchestration in Generative Search

  The paper introduces an iterative orchestration framework that improves generative search recall by dynamically managing evidence allocation and curbing information dilution.

• Optimizing User Representation with Behavioral Densing
  Towards a Densing Law for User Representation Learning at Billion-Scale Capacity

  The researchers introduce a strategy called ALGN to improve user representation learning by reducing redundant behavioral data and optimizing model capacity.

• Efficient Memory for Longer Transformer Sequences
  ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

  ProxyFormer reduces memory overhead by compressing long input sequences into proxy states to allow for significantly larger context processing.

• Interactive World Models with Long Memory
  ReWorld: An Interactive World Model with Long-Horizon Memory

  ReWorld enables interactive video generation with consistent long-term spatial memory by using an efficient chunk-based caching strategy.

• Why AI Agents Change Their Answers
  Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

  This study audits how expanding retrieval corpora causes inconsistency in agent responses even when the model and prompt remain unchanged.

• Efficient Reinforcement Learning for Video Models
  Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

  The researchers introduced a method called OraRL that integrates ground truth annotations as oracle rollouts to improve video model performance and reduce inference latency.

• Efficient Reference Caching for Diffusion Transformers
  Anchoring Instruction Outside Mask: Exact Reference Caching for Efficient In-Context Diffusion Transformers

  The researchers developed a text anchor method to enable high-speed reference image caching in diffusion transformers without sacrificing model performance.

• Managing AI Models with Structured Ontologies
  Ontology-supported AI Model and Dataset Management

  The authors introduce AIMDEP, a platform that uses a specialized ontology to manage AI assets and metadata for improved model tracking and collaboration.

• Memory-Augmented Compression for Efficient Reasoning
  Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning

  The paper introduces a method that improves the efficiency and accuracy of chain of thought reasoning by injecting relevant, pre-computed reasoning patterns into the model prompt.

• Recovering Compressed 4 Bit LLMs
  Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  The paper introduces Quantization-Aware Healing, a practical recipe for recovering compressed 4-bit large language models, and uses it to produce the open-weight model Hypernova-60B.

• Efficient Hybrid Transformer Model for Tabular Data
  Tydra: An Efficient Hybrid Model for Tabular Data

  Tydra combines transformer and state-space architectures to achieve faster inference on tabular data than the existing TabPFN foundation model.