Quantization

What it is

Neural network weights are typically stored as 16-bit floating point numbers. Quantization maps these high-precision values to lower-bit representations, such as 8-bit integers or even 4-bit values. This reduces a model memory footprint by roughly half to three-quarters, allowing you to run a 70-billion parameter model on fewer or smaller GPUs. The underlying mechanism scales and rounds the original weight values into a restricted bucket of integer outputs.

Why it matters

Hardware memory capacity dictates which model sizes you can self-host. If you ignore quantization, you might provision expensive enterprise GPUs with massive VRAM just to serve standard models. Knowing about quantization allows you to host larger models locally or reduce your cloud infrastructure bill by fitting models into cheaper hardware tiers with minimal impact on output quality.

In practice

You encounter quantization when configuring model weights in deployment frameworks like vLLM, Ollama, or llama.cpp, or when downloading GGUF and AWQ files from Hugging Face. You select settings like INT4 or INT8 during model loading. In production, you monitor GPU memory utilization drops alongside minor latency improvements and track benchmark scores to ensure output quality remains acceptable.

The tradeoff

Lower bit-widths reduce memory and latency, but aggressively quantizing below 4 bits often degrades model accuracy and causes noticeable degradation in complex reasoning tasks.

Where it appears

Research summaries that use Quantization, each linked to its source paper.

• Quantization Effects on Bangla Language Understanding
  Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

  The study evaluates how different quantization formats impact the performance of large language models when processing the Bangla language across various natural language understanding benchmarks.

• Efficient Compression of Vision Language Models
  Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs

  The paper introduces S3D8, a quantization format that compresses the Llama 3.2 11B Vision Instruct model to 3.7 GB for mobile CPU execution.

• Verifying Integrity of Vendor Hosted LLMs
  Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

  The paper introduces Ventor-QTest, an audit framework that detects behavioral shifts in third-party LLM APIs by comparing outputs against trusted benchmarks without needing internal model access.

• Reprogramming Models into Assertive Socratic Assistants
  Behavioral Reprogramming of Open-Weights Models: Cognitive Plasticity and Alignment Bounds

  The paper demonstrates how to fine-tune open-weights models to shift from passive assistant behaviors to a proactive Socratic persona using targeted parameter-efficient techniques.

• Optimizing Rotations for 4-Bit Model Quantization
  When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

  The researchers investigated whether using specific rotational transforms that respect RoPE structure improves accuracy during 4-bit model quantization.

• Optimizing Boosted Decision Trees on FPGAs
  FQTree: Fine-grained Quantization and Hardware Generation of Boosted Decision Trees

  The FQTree method uses fine-grained quantization to reduce resource usage while maintaining high accuracy for boosted decision tree models deployed on FPGAs.

• Optimized Compression for Image Models
  HAMP-LIC: Hessian-Aware Mixed-Precision Post-Training Quantization for Learned Image Compression

  The paper introduces HAMP-LIC, a Hessian-aware mixed-precision quantization method that shrinks learned image compression models while preserving image quality and eliminating cross-platform decoding mismatches.

• Improving LLM Quantization Through Reconstructive Rounding
  ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization

  ReRound uses a learned diffusion-based approach to resolve midpoint ambiguity during model quantization, resulting in higher accuracy for compressed LLMs without requiring calibration data.

• Efficient Weight Quantization Using Cubic Curves
  CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights

  CubicQuant introduces a flexible, GPU-friendly weight quantization format that uses monotonic cubic functions to better represent model weight distributions compared to standard uniform methods.

• Accelerating Neural Network Quantization with BaKron
  BaKron: Efficient Quantization with Kronecker-Factored Hessians

  The paper introduces BaKron, a new quantization method that improves efficiency for two-sided Kronecker-factored Hessian approximations in neural networks.

• 8-bit Matrix Multiplication for Large Language Models
  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

  The paper introduces LLM.int8(), a quantization method that enables zero-degradation inference for large language models up to 175B parameters by isolating emergent outlier features into a 16-bit matrix multiplication.

• Efficient Learned Volume Compression for Science
  EVOLVE: Efficient Learned Volume Compression with Variable-Rate Encoding on a Cross-Domain Database

  EVOLVE is a neural volume compression framework that enables variable-rate encoding across diverse scientific datasets using a unified, optimized autoencoder architecture.

• Efficient Memory Management for MoE LLMs
  PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

  PagedWeight manages GPU memory for Mixture-of-Experts models by dynamically quantizing weights at runtime to balance model precision against KV cache requirements.

• Recovering Compressed 4 Bit LLMs
  Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  The paper introduces Quantization-Aware Healing, a practical recipe for recovering compressed 4-bit large language models, and uses it to produce the open-weight model Hypernova-60B.

• Optimizing Neural Architectures for Edge AI
  NAS-Driven Hardware Accelerator Exploration for Edge AI and Quantization Effects on the Pareto Space

  The authors developed a three stage pipeline that integrates automated neural architecture search with hardware aware mapping to mitigate the performance penalties of INT4 quantization.

• Quantizing Financial Forecasting Models
  Calibration Bets on the Past: Post-Training Quantization for Financial Time-Series Forecasting

  The paper investigates how post-training quantization and activation calibration impact cross-sectional volatility forecasting models applied to financial time series.

• Optimizing Softmax Layer Memory in Large Language Models
  SoftWater: Class-Aware Rate Allocation for Softmax Quantization

  The paper introduces SoftWater, a class-aware rate allocation algorithm for quantisation that reduces memory usage in the softmax output layer of Large Language Models.

• Recovering Compressed 4 Bit LLMs
  Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  The paper introduces Quantization-Aware Healing, a practical recipe for recovering compressed 4-bit large language models, and uses it to produce the open-weight model Hypernova-60B.

• Efficient CPU Inference with Daedalus-150M Architecture
  Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference

  The paper introduces Daedalus-150M, a hybrid model combining depthwise convolutions and attention mechanisms to optimize CPU inference efficiency by reducing reliance on key-value cache growth.

• Optimizing Softmax Layer Memory in Large Language Models
  SoftWater: Class-Aware Rate Allocation for Softmax Quantization

  The paper introduces SoftWater, a class-aware rate allocation algorithm for quantisation that reduces memory usage in the softmax output layer of Large Language Models.

• When Multi-Agent Systems Help
  When Do Multi-Agent Systems Help? An Information Bottleneck Perspective

  The paper investigates the performance differences between multi-agent systems and single-agent systems powered by large language models to address why multi-agent advantages vary inconsistently across settings.

• Optimizing Rotations for 4-Bit Model Quantization
  When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

  The researchers investigated whether using specific rotational transforms that respect RoPE structure improves accuracy during 4-bit model quantization.