Large language model

What it is

The model processes inputs by converting text into sequences of tokens, which are numerical representations mapped into high-dimensional space. Through its attention mechanism, it calculates the statistical relationship between these tokens to predict the most likely continuation of a sequence. These models typically contain billions of parameters, requiring significant VRAM for the KV cache during inference. Running these at scale involves balancing GPU memory bandwidth against throughput and latency requirements.

Why it matters

Understanding LLMs allows you to estimate infrastructure costs and latency budgets based on input and output token counts. It changes how you design features, as you must decide between utilizing the model's inherent generalization or building a deterministic pipeline for predictable outputs. Ignoring the probabilistic nature of the model leads to production instability, particularly regarding error handling and rate-limited API calls.

In practice

You interact with these models primarily through APIs by tuning parameters like temperature, top-p, or system instructions to influence output variance. In production, you monitor token usage as a proxy for cost and observe output logs to detect performance degradation or latent hallucinations. You may also adjust quantization settings to reduce memory footprint when deploying on smaller hardware.

The tradeoff

The core tradeoff is between the model's broad general intelligence and its high latency and cost compared to specialized, smaller models or procedural logic.

Where it appears

Research summaries that use Large language model, each linked to its source paper.

• Optimizing LLM Semantic Data Processing Systems
  Compositional Online Learning for Semantic Data Processing Systems

  The paper introduces a layered composition framework for semantic data processing that integrates online learning components to reduce the cost of LLM-based query execution.

• Standardizing Data Generation for AI Agents
  What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents

  The ACE framework establishes a formal structure for evaluating and improving the data generated to train autonomous AI agents.

• Auditing Logic in LLM Data Agents
  Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems

  The paper introduces trace integrity metrics to detect silent failures where LLM data agents produce correct answers through invalid logical steps.

• Dynamic Routing for Efficient LLM Agents
  ProgRouter: Online Progress-Guided Orchestration for Multi-Agent LLM Workflows under Quality-Cost Tradeoffs

  ProgRouter optimizes multi-agent workflows by dynamically selecting models based on progress and cost to maximize task completion rates within defined energy budgets.

• Automating Machine Learning Model Documentation
  Automatic Model Card Generation Using an LLM

  The authors introduce two systems, MCTidy and MCGenie, that use large language models to automatically reorganize and generate standardized documentation for machine learning models.

• Benchmarking Resource Aware LLM Tool Agents
  PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents

  The paper introduces PeakBench to evaluate how effectively AI agents manage system resources when executing parallel tool workflows.

• Building Industrial Datasets from Technical Reports
  Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports

  The authors present an end-to-end framework to automatically generate instruction-tuning and benchmark datasets from complex industrial technical documents.

• Detecting Fake Recommendations in LLMs
  One Polluted Page Is Enough: Evaluating Web Content Pollution in LLM Recommenders

  Researchers developed a benchmark called FORGE to measure how easily LLMs can be tricked into recommending fake products through search-augmented content.

• Automating Patent Drafting from Inventor Disclosures
  Benchmarking Patent Drafting from Inventor-Style Disclosures

  The authors introduce a multi-agent system called Patent-MAF that processes raw invention disclosures into formal patent specifications and claims.

• Structuring Entities for Better Document Retrieval
  EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering

  EnSI-RAG improves long-document question answering by indexing documents based on structured entity relationships rather than simple text chunks.

• Securing Autonomous AI Agents with ClawSentry
  ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents

  ClawSentry provides a modular, multi-tier security framework that uses an abstraction protocol to protect autonomous LLM agents against progressive execution threats.

• How Coding Agents Use Technical Documentation
  From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation

  This study analyzes how autonomous coding agents interact with documentation through an empirical examination of their file-level changes and conversational logs.

• Improving AI Models for Chemical Synthesis
  Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis

  The researchers developed an LLM that uses Reinforcement Learning Fine-Tuning and specialized rewards to improve performance in single-step retrosynthesis tasks.

• Improving Software Agents With Self-Distilled Knowledge
  SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution

  SkillForge enhances software engineering agents by distilling repository-specific knowledge into reusable skills to solve project-specific issues.

• Agentic 3D Creation via Domain Specific Languages
  aDSL: Agentic 3D Creation via Joint Agent-Program Design

  The paper introduces aDSL, a domain-specific language and agent-based system designed to reliably convert natural language instructions into functional 3D programs and geometry.

• Managing Cognitive Risks in Agentic AI
  Understanding Cognition-Induced Risks in Agentic AI Systems

  The paper defines a framework to categorize cognitive risks in agentic AI and proposes mitigation strategies to maintain human control over autonomous systems.

• Automated Security Annotation for Business Processes
  A Hybrid LLM-Based Framework for Automated Security Annotation Generation in Business Process Models

  The paper introduces a hybrid LLM-based framework that automates the generation of SecBPMN2 security annotations from natural-language specifications to improve process model accuracy.

• Building Realistic Mental Health Simulation Agents
  PatientAct: Theory-Grounded Mental Health Client Simulation

  The paper introduces PatientAct, a framework designed to simulate mental health clients using clinical theory to improve realism in LLM-based agent interactions.

• Why AI Judges Change Their Verdicts
  Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence

  The researchers created a framework to test if AI models acting as judges maintain consistent opinions when subjected to adversarial pressure or repetitive questioning.

• Enforcing Contract Compliance for Automated Proofs
  CAPRI: Contract-Aware Proof Repair for Isabelle

  The paper introduces a contract-based verification system that prevents LLM-powered proof repair tools from making unauthorized edits to protected code in the Isabelle assistant.

• Building Reliable Clinical AI with Multi-Agent Systems
  MARC v1: An Open-Source Multi-Agent Framework for Clinical AI Reasoning and Coordination

  MARC v1 replaces monolithic LLM prompting with a deterministic multi-agent framework to improve clinical reasoning and enable step by step error tracking.

• A New Benchmark for Detecting Vulnerable Code
  VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

  VICBench is a new multi-language benchmark containing 100 verified vulnerability-inducing commits that helps evaluate automated security detection tools.

• Improving Retrieval Accuracy with SQL Hyperedges
  SAG: SQL-Retrieval Augmented Generation with Query-Time Dynamic Hyperedges

  The paper introduces SAG, a retrieval system that improves RAG performance by using SQL joins to dynamically discover cross-document associations through event-based hyperedges.

• Benchmarking Triton Kernel Generation for AI
  RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

  The authors introduce RealisticTritonBench to evaluate LLM performance in generating production-grade Triton kernels for real-world AI frameworks.

• Hijacking LLM Agents for Resource Amplification
  Convergent Detour Hijacking: Task-Preserving Resource Amplification in Skill-Based LLM Agents

  The paper introduces a method called Convergent Detour Hijacking that steers LLM agents into unnecessarily costly execution paths while preserving the final task output.

• Automated Security Testing for LLM Agents
  ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

  ToolHazard provides a scalable framework to automatically synthesize stateful environments and generate adversarial tasks for evaluating and aligning LLM-based agents.

• Automated Taxomony Expansion Using LLMs
  ReLTEx: Reliable LLM-based Taxonomy Expansion

  ReLTEx improves automated taxonomy expansion by using LLMs for candidate generation combined with a structure-aware classifier to ensure hierarchical consistency.

• Reducing Bias in LLM Stance Measurement
  Templated or fully Synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance

  The paper demonstrates that using templated prompts creates structural artifacts that bias LLM political stance measurements, whereas LLM-generated prompts produce more realistic and neutral results.

• Benchmarking AI Agents in Dynamic Life
  VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

  The paper introduces VibeLifeBench, a framework for evaluating agent performance in long-term, multi-week scenarios that involve state persistence and silent environment changes.

• Optimizing LLMs for Better Performance and Efficiency
  Can We Optimize the Performance-Carbon Emission Break-Even Point?: The Quest for Greener LLMs

  The paper introduces a joint loss optimization technique that fine-tunes LLMs to improve task accuracy while simultaneously minimizing inference carbon emissions.

• LLM Verification Layers for Robot Autonomy
  Agentic Harnesses: LLM-Driven Verification Layers for Robot Autonomy

  The paper introduces a multi-layered verification framework that uses an ensemble of LLM judges to validate robot action plans for safety, security, and ethical alignment before execution.

• Selecting Better CAD Programs Through Consensus
  Test-Time Scaling for CAD Generation via Verifier-Free Consensus Selection

  The paper introduces a verifier-free consensus selection method that improves the geometric accuracy of parametric CAD programs generated by language models.

• Automated Safety Harness Evolution for Agents
  SHE: Trajectory-driven Safety Harness Evolution for LLM Agents

  The SHE framework allows LLM agent safety systems to automatically evolve over time by analyzing failure trajectories to refine safety boundaries.

• Improving Multi-Agent Systems through Evidence-Guided Revision
  EMAS: Stabilizing Multi-Agent System Evolution through Evidence-Guided Revision

  EMAS evolves the topology and prompts of multi-agent systems using evidence-based revisions to increase accuracy and reduce token costs while keeping the base language model fixed.

• CAS2UML: A Sketch to Code Dataset
  CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

  The authors created a dataset of 557 hand-drawn UML diagram pairs paired with verified, machine-readable PlantUML code to support automated diagram generation.

• Benchmarking Personality Evolution in AI Agents
  Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events

  Researchers evaluated how LLM agents change their personality traits in response to life events using a new benchmark called BFI-Adapt.

• Turning LLM Repairs into Reusable Memory
  From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL

  The paper introduces a method to store verified text-to-SQL repair episodes as a reusable memory bank that improves performance on future questions over the same database.

• Automated Stress Testing for Vision Models
  SABRE: Scalable and Automated Benchmarking of VLMs under Stress

  The paper introduces SABRE, a scalable, automated pipeline that generates challenging stress tests to expose weaknesses in how vision-language models reconcile visual evidence with existing world knowledge.

• Improving LLM Reliability for Scientific Data Analysis
  Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

  Fisher-R1 is a specialized LLM agent trained to perform reliable hypothesis testing by using a new benchmark and outcome-grounded reinforcement learning.

• Mapping Open Source AI Risk Tools
  Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools

  The paper classifies 21 open-source AI security tools against the MIT AI Risk Mitigation and Response Taxonomy to identify gaps in existing risk coverage.

• Optimizing Agentic Context via Blast Radius
  Blast Radius

  Blast Radius reduces LLM token consumption by identifying and archiving redundant or concluded context in agentic coding environments.

• Standardizing LLM Router Development and Deployment
  LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

  LLMRouter provides a unified framework and automated data pipeline to build, evaluate, and deploy routers that select the most cost-effective LLM for a given task.

• Improving AI Terminal Task Construction
  CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

  CalibForge uses automated adversarial feedback from software solvers to ensure that training tasks for LLM agents are neither too simple nor impossible to solve.

• Deterministic Activity Compilation for Agent Memory
  Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

  The paper introduces a deterministic method to compile raw screen activity into structured, auditable memory frames for computer-use agents.

• Automated Construction of Relational Data Schemas
  Tytan: Interactive Neurosymbolic Construction of Analytic Semantic Schemas from Relational Data

  Tytan uses neurosymbolic AI to automatically build semantic schemas from raw relational databases by combining LLM-driven inference with deterministic verification.

• Building Reusable Skills for Coding Agents
  Learning Globally Reusable Skills for Coding Agents

  The paper introduces a framework to evolve agent skills as an interconnected global system rather than isolated updates to improve performance and generalizability.

• Coach Models Improve Language Model Training
  LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

  The paper introduces an LLM-as-a-Coach method that replaces traditional scalar rewards in reinforcement learning with rich experiential knowledge to train open-ended task models.

• Audio-Visual Flamingo for Complex Videos
  Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

  The paper introduces Audio-Visual Flamingo, an open model designed to improve joint perception, temporal alignment, and multi-event reasoning over long videos.

• WanSong v1.0 High Fidelity Audio Generation
  WanSong v1.0 Technical Report

  WanSong v1.0 is a music generation model that uses a hybrid transformer architecture and a dual-stem output strategy to improve the fidelity and separation of vocals and background music.

• Contextual Human Aligned AI Evaluation Workflow
  Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications

  Project Kaleidoscope introduces a workflow for calibrating automated LLM judges against human labels to evaluate real world AI applications.

• Structuring Agent Harnesses for Better Edits
  Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

  The paper introduces the Harness Handbook, a behavior-centric documentation system that helps agents and developers navigate and modify large, complex agent codebases.

• Detecting Malicious NPM Packages Using Agents
  ProfMalPlus: Agent-Coordinated Detection of Malicious NPM Packages via Static-Dynamic Analysis Synergy

  ProfMalPlus uses a multi-agent reasoning framework to detect malicious NPM packages by combining static code analysis with dynamic verification.

• Securing Large Language Model Agent Skills
  Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation

  The researchers developed a security framework to protect the lifecycle of reusable LLM agent skills from creation through execution.

• Improving LLM Accuracy with Targeted Critiques
  CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes

  CritICL improves large language model performance by providing targeted, failure-aware critique examples in the prompt to steer reasoning away from common pitfalls.

• Decoupling Persona and Execution for Agents
  Persona-Execution Separation: An Architecture Pattern for Evolving LLM Agents under Execution Audit

  The paper introduces a dual-domain architectural pattern that separates an AI agent's persona from its execution logic to improve governance and auditability in regulated environments.

• Debugging LLM Multi-Agent System Failures
  Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

  The authors introduce SymTrace, a framework that improves the reliability of debugging complex multi-agent systems by using controlled intervention anchors to replicate and repair execution failures.

• Prior Scores Bias LLM Evaluation Results
  Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

  The research demonstrates that providing prior evaluation scores as metadata to LLM-as-a-judge systems causes systematic anchoring bias that distorts final judgment accuracy.

• Benchmarking LLM Unit Test Generation
  XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models

  The researchers introduced XRepoTest to evaluate how effectively large language models generate unit tests within complex, multi-file code repositories across five programming languages.

• Faster LLM Reasoning via Prefix Sliding
  Prefix Sliding for efficient test-time scaling

  Prefix Sliding enables large language models to perform reasoning tasks three times faster without requiring additional training.

• Optimizing LLM Agents for Competitive Programming
  MARS: Multi-Specialist LLM Relay System for Competitive Programming

  The Multi-Specialist LLM Relay System improves coding agent performance by using specialized agents and a staged pipeline to solve complex programming tasks.

• Linking Code and Prose for Coding Agents
  A Literate Programming Environment for Human and Machine Agents

  The paper introduces notlob, a system that maintains a graph of prose and code to improve how coding agents navigate and manage development context.

• Auditing Medical AI Reasoning Reliability
  Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought

  Researchers evaluated how medical Large Language Models reason by systematically perturbing inputs and measuring if the model's logical chain stays consistent with its final output.

• Automated Benchmark for Culinary Language Models
  FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth

  FlavourBench is an automated evaluation framework that uses the Epicure culinary system to rank language models on objective, executable tasks.

• Fixing Memory Bias in LLM Agents
  MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

  The paper introduces MemTrapBench to quantify how retrieved memory degrades LLM reasoning and proposes AdaptiveMem as a solution to prevent memory-induced cognitive traps.

• Reliable PLC Code Generation With Agents
  SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

  SemaPLC uses an agent-based workflow and multi-stage verification to ensure that LLM-generated PLC programs integrate correctly and execute reliably within existing industrial projects.

• Benchmarking Cognitive Bias in LLMs
  AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

  The paper introduces AnchorBench to evaluate the anchoring effect in large language models across multiple pathways and relevance conditions.

• Lowering Inference Costs Using Speculative Decoding
  SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

  SPADE lowers inference costs and latency for large language models by using an edge-based draft model to generate token sequences that are verified by a cloud-based model.

• Automating Complex System Diagnostics With Knowledge Graphs
  Constructing Dynamic Master Logic Models as Knowledge Graphs for Complex System Diagnostics Using Retrieval-Augmented Large Language Models

  The authors developed an automated framework using Retrieval-Augmented Generation and Large Language Models to construct Dynamic Master Logic models as Knowledge Graphs, overcoming the scalability limits of manual expert interpretation.

• Understanding Massive Activations in Hybrid Large Language Models
  Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

  The paper tracks massive activation tokens across hybrid linear attention large language models to understand how layerwise hybridization reshapes internal activation dynamics.

• Evaluating LLM Reliability in SPICE Netlist Tasks
  NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

  The paper introduces NetlistBench, a benchmark containing 2,342 cases across 24 task families to evaluate large language model reliability in recognizing and manipulating simulator-facing SPICE netlists.

• Unified Pixel and Language Medical Model
  MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

  MedPixel combines visual reasoning and image segmentation into a single architecture to bridge the gap between clinical text and pixel-level data.

• Evaluating LLMs as Mobile Personal Assistants
  SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information

  SPIEval is a new human-curated benchmark designed to evaluate how effectively large language models handle complex tasks using personal information scattered across mobile applications.

• Self-Evolving Code Generation Agents
  Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution

  Ouroboros introduces a self-developing agent architecture that treats code evolution as a task to adapt tools and core logic autonomously while maintaining strict safety standards.

• Benchmarking LLM Unit Test Generation
  XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models

  The researchers introduced XRepoTest to evaluate how effectively large language models generate unit tests within complex, multi-file code repositories across five programming languages.

• Quantization Effects on Bangla Language Understanding
  Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

  The study evaluates how different quantization formats impact the performance of large language models when processing the Bangla language across various natural language understanding benchmarks.

• Securing Large Language Models with Semantic Overlays
  Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

  Semantic Overlays add an out-of-band annotation layer to input streams to help models distinguish between trusted instructions and untrusted content, significantly reducing prompt injection success rates.

• Why Large Language Models Miss Information
  Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows

  This paper shows that while models can retrieve relevant financial data from long documents, they fail to act on that information unless the data is restated specifically for the decision prompt.

• Benchmarking Explicit Strategy Induction in LLMs
  StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

  The paper introduces StrategyBench to evaluate if language models can effectively derive and apply explicit task-level strategies from few-shot examples.

• Improving Rule Reasoning in Large Models
  Beyond Factual Knowledge: Benchmarking and Learning Step-Level Procedural Rule Reasoning in Large Language Models

  The paper introduces a framework called DynaRule that enables large language models to dynamically retrieve and apply reusable procedural rules at scale.

• Improving Multimodal Model Safety Through Reframing
  ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models

  ReFrame acts as a secure intermediary layer that analyzes and rewrites potentially unsafe multimodal inputs before they reach closed source models.

• Improving AI Agents for Software Engineering
  Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda

  The paper provides a structured survey and research agenda to bridge the gap between software engineering and security when evaluating Large Language Model agents.

• Improving Structured Pruning for LLMs
  COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

  The COEC method improves the accuracy of pruned large language models by using a specialized, two-sided rotation technique that avoids the pitfalls of direct weight refitting.

• Improving AI Models for Chemical Synthesis
  Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis

  The researchers developed an LLM that uses Reinforcement Learning Fine-Tuning and specialized rewards to improve performance in single-step retrosynthesis tasks.

• How Language Models Choose Between Sources
  When Text and Numbers Disagree: Evidence Arbitration in Large Language Models

  Researchers evaluated how large language models weigh conflicting evidence, finding they often over-rely on numerical data and external tool forecasts.

• Benchmarking Selective Knowledge Removal in LLMs
  ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

  The authors introduce ConceptGuard, a benchmark that evaluates how effectively LLMs can remove harmful knowledge while retaining benign functionality for dual-use concepts.

• Improving Robot Planning With Rule-Compliant Models
  Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models

  The researchers developed a framework that improves how multimodal large language models follow natural-language rules when performing spatial planning tasks.

• Controlling LLM Reasoning Costs via Self-Reflection
  Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models

  The paper introduces a training-free inference-time protocol that uses a self-critique loop and a confirmed sentinel to improve reasoning accuracy while early-stopping redundant computations.

• Automated Analog Circuit Design Using LLMs
  AaLLM: An End-to-End Analog Circuit Design Framework from Topology Generation to Sizing Using Large Language Models

  AaLLM is an end-to-end framework that uses a chain of LLM agents to automatically generate and size analog circuits from user specifications.

• Solving Logical Reasoning Failures in Large Language Models
  From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

  The paper introduces a framework that decomposes compound answer options into atomic statements to help language models correctly evaluate explicit logical operators like And, Or, and Neither/Nor.

• Enhancing Video Reasoning in Multimodal Models
  Motion-as-Prompt: Enhancing Motion Reasoning in Multimodal Large Language Models via Motion-Guided Cross-Frame Visual Prompting

  The paper introduces a method called Motion-as-Prompt to improve multimodal large language models by adding motion-guided visual markers to video frames.

• Optimizing Softmax Layer Memory in Large Language Models
  SoftWater: Class-Aware Rate Allocation for Softmax Quantization

  The paper introduces SoftWater, a class-aware rate allocation algorithm for quantisation that reduces memory usage in the softmax output layer of Large Language Models.

• Removing Person Knowledge from Multimodal Models
  PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models

  The paper introduces a corpus-free framework and benchmark for deleting specific person-related knowledge from multimodal large language models without needing the original training data.

• Mapping Behavioral Evolution in Large Models
  Mapping and Measuring the Behavioral Evolution of Large Language Models

  The researchers developed a method to quantify how different language models behave and evolve by measuring distances between their responses to a shared set of 10,000 prompts.

• Detecting LLM Hallucinations via Attention Path Fragility
  Attention-Path Fragility as an Uncertainty Signal in Large Language Models

  The paper introduces a technique called ASMI that measures model uncertainty by observing how responses change when random paths in the transformer's attention mechanism are disrupted.

• Improving Multilingual Translation Without Reference Data
  Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation

  The researchers developed a reference-free post-training method to optimize machine translation models using only source-side text, bypassing the need for high-quality parallel data.

• Benchmarking LLMs on Evolving Regulatory Documents
  Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding

  The paper introduces a new benchmark to evaluate how well large language models can resolve specific versions of legal documents that change over time.

• Evaluating Large Language Models for Government
  From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

  Researchers developed a new, systematic evaluation suite to help Dutch government agencies select and deploy large language models based on transparency, honesty, and operational efficiency.

• Optimizing Reasoning and Response Modes in LLMs
  Fusion Training for Mathematical Generalization in Large Language Models

  The researchers investigated how to balance a model's ability to provide concise answers with its capacity for long-form mathematical reasoning by testing different training schedules and data ratios.

• Testing LLM Spatial Reasoning and Mapping
  Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

  Researchers evaluated how effectively large language models understand and manipulate spatial relationships like direction, distance, and topology using a custom geographic benchmark.

• Measuring Political Bias in Large Language Models
  Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

  Researchers developed a systematic framework to audit and measure how large language models exhibit political bias when analyzing international conflicts.

• Using AI to Reduce Belief in Conspiracy Theories
  Reducing belief in conspiracy theories as they unfold using large language models

  The researchers evaluated if multi-turn LLM conversations can effectively debunk conspiracy theories as they emerge during crisis events.

• Building Autonomous Agents for Minecraft
  Voyager: An Open-Ended Embodied Agent with Large Language Models

  Voyager is an embodied agent that uses GPT-4 to autonomously learn and store executable code skills for complex task completion in Minecraft.

• Tree Search for Language Models
  Tree of Thoughts: Deliberate Problem Solving with Large Language Models

  The paper introduces a framework called Tree of Thoughts that frames problem-solving as a search over a tree of partial solutions, allowing language models to perform deliberate exploration, lookahead, and backtracking.

• 8-bit Matrix Multiplication for Large Language Models
  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

  The paper introduces LLM.int8(), a quantization method that enables zero-degradation inference for large language models up to 175B parameters by isolating emergent outlier features into a 16-bit matrix multiplication.

• Finding the Optimal Size for Language Models
  Training Compute-Optimal Large Language Models

  The paper investigates the optimal model size and number of tokens for training a transformer language model under a given compute budget, finding that current large language models are significantly undertrained.

• Evaluating Large Language Models Trained on Code
  Evaluating Large Language Models Trained on Code

  The paper investigates the program synthesis and code-writing capabilities of large language models trained on code, focusing on generating standalone Python functions from docstrings and measuring functional correctness.

• Teaching Large Language Models To Reason
  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

  The paper introduces chain-of-thought prompting, a technique where models generate intermediate reasoning steps before answering, which significantly improves their performance on complex reasoning tasks.

• Efficient Fine-Tuning for Large Language Models
  LoRA: Low-Rank Adaptation of Large Language Models

  The paper introduces Low-Rank Adaptation, a method that freezes pre-trained model weights and trains low-rank decomposition matrices to update models efficiently without introducing inference latency.

• Efficient Table Reasoning Through Partitioned Analysis
  PARTAB: Partition-Aware Reasoning with Structured Evidence for Scalable Table Understanding

  PARTAB improves table-based reasoning by decomposing tables into semantically coherent parts before processing them with a multi-stage pipeline.

• Automated Assessment of AI Project Readiness
  RAIL: An Automatic Classifier of the Artificial Intelligence Readiness Level

  The paper introduces RAIL, a multi-agent framework that uses independent LLM experts to objectively evaluate the readiness level of AI projects and prevent maturity inflation.

• Efficient Data Selection for LLM Training
  PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning

  PPL-Factory is a method that selects the most informative subset of training data to improve fine-tuning performance while reducing computational overhead.

• Jointly Training AI to Create and Use Tools
  Joint Optimization of Tool Creation and Use for Large Language Model Agents

  The SMITH framework uses reinforcement learning to train a single language model to both synthesize reusable Python tools and apply them effectively to solve complex tasks.

• How AI Models Choose Doctors for Patients
  Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

  Researchers audited seven large language models to determine how they weigh physician reputation and demographic factors when recommending doctors to patients.

• Measuring Political Bias in Large Language Models
  Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

  Researchers developed a systematic framework to audit and measure how large language models exhibit political bias when analyzing international conflicts.

• Using Language Models for Metabolite Analysis
  MetaboLLM: a metabolomics-specialized large language model for biochemical knowledge integration and predictive metabolite graph construction

  Researchers developed MetaboLLM to integrate biochemical knowledge and convert it into predictive metabolite graphs for clinical diagnostics.

• Securing Large Language Model Agent Skills
  Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation

  The researchers developed a security framework to protect the lifecycle of reusable LLM agent skills from creation through execution.

• Improving AI Memory Evaluation in Conversation
  MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

  The paper introduces MemUse, a benchmark for evaluating how well conversational AI integrates long-term memory into natural dialogue, revealing a significant disconnect between standard fact-checking performance and actual conversational utility.

• Benchmarking Mobile Agent Planning Capabilities
  MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

  MobilePA-Bench is a stateful, tool-centric benchmark environment designed to evaluate how well mobile planner agents handle complex, multi-step tasks.

• How Emotional Context Triggers Model Sycophancy
  Affective Context Amplifies Sycophancy in LLM Responses

  The paper demonstrates that LLMs become significantly more sycophantic when responding to users expressing specific emotional states.

• How Token Budget Changes LLM Rankings
  Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

  The paper demonstrates that model rankings shift significantly based on the token generation budget allowed during inference, challenging the reliability of standard static evaluation benchmarks.

• Privacy Preserving Alzheimer's Detection via Speech
  LSEAD: A Privacy-Preserving LLM-Based Speech Analysis Framework for Early Alzheimer's Disease Screening

  The LSEAD framework provides a cost effective method for early Alzheimer's detection by analyzing speech through locally hosted language models.

• Testing How Well LLMs Judge Responses
  A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

  The paper formalizes construct validity for LLM-as-a-judge evaluation and reveals that current judges are significantly more sensitive to scope edits than strength edits.

• Teaching LLMs Clinical Reasoning for ICUs
  Teaching LLMs How ICU Physicians Approach Clinical Reasoning Through OMOP-Aligned Retrieval Improves Reasoning Across Clinical Domains

  The authors created the ICU-REACT dataset and a corresponding family of fine-tuned models to improve LLM performance in identifying and reasoning over patient data for critical care.

• Benchmarking Explicit Strategy Induction in LLMs
  StrategyBench: Evaluating Explicit Strategy Induction in Large Language Models

  The paper introduces StrategyBench to evaluate if language models can effectively derive and apply explicit task-level strategies from few-shot examples.

• Improving Stable Critic Training for LLMs
  How to Train a Critic Stably and Efficiently

  The paper introduces BPCO, a method that stabilizes critic-based reinforcement learning, improving performance across various model sizes and tasks.

• Detecting Fake Recommendations in LLMs
  One Polluted Page Is Enough: Evaluating Web Content Pollution in LLM Recommenders

  Researchers developed a benchmark called FORGE to measure how easily LLMs can be tricked into recommending fake products through search-augmented content.

• Steering LLM Psychotherapy Interactions
  Move by Move: Measuring and Steering How LLMs Conduct Psychotherapy

  The authors introduce a method to steer LLM behavior in therapy sessions by exposing a clinical move ontology as tools, which improves alignment with human therapists.

• Recovering Compressed 4 Bit LLMs
  Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  The paper introduces Quantization-Aware Healing, a practical recipe for recovering compressed 4-bit large language models, and uses it to produce the open-weight model Hypernova-60B.

• Enforcing Privacy Policies in LLMs via Attention
  Personalized Privacy Control in LLMs via Attention Head Intervention

  The paper introduces a method to improve privacy policy adherence in LLMs by intervening on specific attention heads to align model outputs with user-defined privacy preferences.

• Improving Structured Pruning for LLMs
  COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models

  The COEC method improves the accuracy of pruned large language models by using a specialized, two-sided rotation technique that avoids the pitfalls of direct weight refitting.

• Benchmarking Real Time Omni Modal Assistants
  OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

  OmniAssistBench is a new evaluation framework designed to measure how well multimodal AI models handle complex, multi-turn interactions in real-time video scenarios.

• Evaluating Legal Advice Accuracy in LLMs
  InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

  InsufficiencyBench measures how effectively LLMs identify missing information in legal queries instead of providing premature, potentially fabricated advice.

• Evaluating Cache Eviction Policies for LLMs
  Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

  The paper systematically evaluates various eviction policies for semantic LLM caching and finds that the standard Least Frequently Used approach remains highly effective compared to more complex alternatives.

• Benchmarking Selective Knowledge Removal in LLMs
  ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

  The authors introduce ConceptGuard, a benchmark that evaluates how effectively LLMs can remove harmful knowledge while retaining benign functionality for dual-use concepts.

• Optimizing GPU Kernels Using LLMs
  PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX

  The authors introduce PTXBench and a supervised fine-tuning method to help LLMs write efficient architecture-specific GPU code.

• Safely Shaping Agent Rewards with LLMs
  Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

  The paper demonstrates that using LLM progress scores to shape rewards for reinforcement learning agents does not alter the underlying optimal policy of the agent.

• Verifying Integrity of Vendor Hosted LLMs
  Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

  The paper introduces Ventor-QTest, an audit framework that detects behavioral shifts in third-party LLM APIs by comparing outputs against trusted benchmarks without needing internal model access.

• Benchmarking Cognitive Bias in LLMs
  AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

  The paper introduces AnchorBench to evaluate the anchoring effect in large language models across multiple pathways and relevance conditions.

• Boosting Parallel Decoding in Diffusion LLMs
  CForce: Boosting Parallel Decoding for dLLMs via Consistency Forcing

  CForce improves parallel text generation in diffusion large language models by using later decoding stages to stabilize predictions in earlier stages.

• Decoupling Knowledge and Reasoning for LLMs
  Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

  The paper introduces Intern-S2-Mobius, an architecture that decouples knowledge storage from reasoning to achieve faster inference speeds while maintaining high performance.

• How Linguistic Style Influences LLM Response Quality
  It's How You Ask: Gender-Associated Linguistic Bias in LLMs

  The paper investigates whether prompts containing linguistic features associated with women negatively impact the quality of responses generated by large language models.

• Do Language Models Know Their Limits
  Toward a Gricean Retreat: Probing LLMs for Knowledge Boundaries and Referent Specificity

  The researchers evaluated whether large language models can intentionally provide less specific answers when they encounter entities outside their training knowledge to avoid hallucinations.

• Automated Analog Circuit Design Using LLMs
  AaLLM: An End-to-End Analog Circuit Design Framework from Topology Generation to Sizing Using Large Language Models

  AaLLM is an end-to-end framework that uses a chain of LLM agents to automatically generate and size analog circuits from user specifications.

• Benchmarking LLMs on Scientific Diagram Tasks
  Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

  The paper introduces Diagram-MMU, a benchmark evaluating how effectively Multimodal Large Language Models handle scientific diagram parsing, editing, and question answering.

• Curated RAG Systems Outperform Frontier LLMs
  A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

  Researchers developed VITA, a domain-specific retrieval-augmented generation system that achieves superior performance on clinical tasks by prioritizing curated local data over generic model scale.

• Extracting Structured Knowledge from Language Models
  REAP: Relation-Aware Elicitation and Parsing for Closed-Book Knowledge Base Construction from LLMs

  The REAP system improves closed-book knowledge base construction by using a two-stage process of relation-aware prompting and hybrid parsing to extract structured data from large language models.

• Automated Taxomony Expansion Using LLMs
  ReLTEx: Reliable LLM-based Taxonomy Expansion

  ReLTEx improves automated taxonomy expansion by using LLMs for candidate generation combined with a structure-aware classifier to ensure hierarchical consistency.

• Evaluating LLMs as Mobile Personal Assistants
  SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information

  SPIEval is a new human-curated benchmark designed to evaluate how effectively large language models handle complex tasks using personal information scattered across mobile applications.

• Benchmarking LLMs on Evolving Regulatory Documents
  Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding

  The paper introduces a new benchmark to evaluate how well large language models can resolve specific versions of legal documents that change over time.

• Improving Complex Question Answering with Graphs
  KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs

  KGCaRe enhances complex conditional reasoning in language models by combining structured knowledge graph lookups with traditional document retrieval.

• Optimizing LLMs for Better Performance and Efficiency
  Can We Optimize the Performance-Carbon Emission Break-Even Point?: The Quest for Greener LLMs

  The paper introduces a joint loss optimization technique that fine-tunes LLMs to improve task accuracy while simultaneously minimizing inference carbon emissions.

• Optimizing Reasoning and Response Modes in LLMs
  Fusion Training for Mathematical Generalization in Large Language Models

  The researchers investigated how to balance a model's ability to provide concise answers with its capacity for long-form mathematical reasoning by testing different training schedules and data ratios.

• Evaluating Theory of Mind in LLMs
  Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics

  The paper introduces Avalon-ToM-Bench, a new benchmark designed to measure how well Large Language Models understand human mental states using the mechanics of the game The Resistance: Avalon.

• Exposing Hidden Reasoning Traces in LLMs
  Stealing Reasoning Traces from Proprietary LLM APIs

  Researchers discovered an architectural flaw in how major LLM providers handle encrypted reasoning traces, allowing them to decrypt and expose proprietary data.

• GeoBenchLLM: Evaluating Geographic Intelligence in LLMs
  GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

  The authors introduce GeoBenchLLM, a unified benchmark consisting of 421,041 questions across twelve datasets to evaluate LLM performance on geographic tasks.

• Reading ESG Concept Content from LLMs
  Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes

  The paper demonstrates that monitoring internal activations of frozen LLMs can measure ESG concept content in text as an alternative to fine-tuning.

• Improving LLM Creativity While Maintaining Alignment
  CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

  The CreativeInstruct method introduces a way to fine-tune a single unified model that balances instruction following with narrative diversity by tagging and self-injecting creative text segments.

• Automating Visual Token Pruning for LLMs
  An AI4AI Framework for Visual Token Pruning

  The paper introduces AutoPrune, a framework that uses large language models to automatically design efficient algorithms for reducing the number of visual tokens in multimodal models.

• Mechanistic Safety Exploits in Diffusion LLMs
  Diffusion LLMs as Targets and Adversaries: Mechanistic Safety Exploits

  The paper demonstrates how diffusion large language models have structural safety vulnerabilities that can be exploited using safety neuron identification and targeted steering techniques.

• Teaching Logic Before Language to LLMs
  Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

  Researchers improved language model learning efficiency by pre-pretraining a Transformer backbone on formal logic derivation sequences before standard language training.

• Incentivizing LLM Reasoning via Pure Reinforcement Learning
  DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

  The paper introduces DeepSeek-R1, demonstrating that large language models can develop advanced reasoning capabilities through pure reinforcement learning without requiring prior supervised fine-tuning.

• Verifying Step by Step Reasoning in LLMs
  Let's Verify Step by Step

  The paper compares outcome supervision with process supervision for training large language models to reduce logical mistakes and hallucinations during complex multi-step reasoning tasks.

• GPT-3 Few-Shot Language Learning Performance
  Language Models are Few-Shot Learners

  The paper introduces a 175-billion parameter model capable of performing tasks with zero or few examples provided in the prompt without needing model weight updates.

• Training Deep Bidirectional Language Models
  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  This paper introduces BERT to enable pre-training of deep bidirectional representations by jointly conditioning on both left and right context in all layers.

• Interleaving Reasoning and Acting in LLMs
  ReAct: Synergizing Reasoning and Acting in Language Models

  The ReAct paradigm enables language models to solve complex tasks by interleaving verbal reasoning with external API actions to dynamically update plans.

• Benchmarking LLMs for 3D Molecular Design
  Do Language Models Dream of Binding Molecules? Benchmarking LLMs under Spatial Constraints

  The paper introduces the 3D-Fit benchmark to evaluate how effectively general-purpose LLMs navigate spatial constraints in molecular generation compared to specialized diffusion models.

• Tuning LLMs for Better Arithmetic Performance
  Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies

  This paper investigates why large language models struggle with basic arithmetic and proposes human-inspired strategies, including task decomposition and specific prompting techniques, to improve their performance.

• Evaluating Open-Weight LLMs for Threat Data Generation
  Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

  The paper evaluates open-weight Large Language Models on converting unstructured connected and autonomous vehicle vulnerability descriptions into structured threat information expressions.

• Efficient Memory Management for MoE LLMs
  PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

  PagedWeight manages GPU memory for Mixture-of-Experts models by dynamically quantizing weights at runtime to balance model precision against KV cache requirements.

• Expanding LLM Tokenizers Without Discarding Weights
  In-Place Tokenizer Expansion for Pre-trained LLMs

  This paper presents an in-place recipe to expand pre-trained model tokenizers to better support underrepresented languages without discarding existing model weights.

• Recovering Pruned LLM Generation Capabilities
  ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

  ShortOPD uses a dynamic distillation strategy to fix structural collapse in pruned LLMs by adjusting training rollouts based on model output quality.

• Replacing Recurrence with Parallel Attention Mechanisms
  Attention Is All You Need

  The Transformer architecture replaces sequential neural networks with an attention based mechanism that enables full parallelization during training.