Back to Feed
Efficiency & Inference / Benchmarks & Evals

Memory Efficient Optimization for Model Adaptation

Original: Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
By Junming Zhang, Shuyu Yin, Peilin Liu, Rendong Ying, Fei Wen Published 2026-08-12

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 1 concepts

Key Takeaways

  • CAZO achieves a substantial reduction in memory usage, requiring only 1,695 MB compared to 6,404 MB for the TENT baseline.
  • The method reaches an average accuracy of 69.0 percent on ImageNet-C at severity level 5 under the standard model reset protocol.
  • In continual adaptation scenarios without model reset, CAZO maintains an average accuracy of 65.3 percent.
  • By using zeroth order optimization, the method eliminates the need for backpropagation, which is the standard mechanism for updating model weights.

Summary & Methodology Analysis

Test-time adaptation (TTA) improves model performance on shifting data domains by tuning parameters on unlabeled samples. Standard approaches rely on backpropagation (BP), which calculates gradients by propagating error signals backward through the network. This process is memory intensive because it requires storing intermediate activations for the entire model. CAZO bypasses this by utilizing zeroth-order (ZO) optimization, which approximates gradients through forward-only function evaluations. Traditional ZO methods suffer from high variance when sampling random vectors for these approximations, making them less reliable than first-order methods. CAZO mitigates this by leveraging a sliding-average estimation of the diagonal Hessian, which represents the local curvature of the loss surface. By constructing a covariance matrix based on this curvature, the method performs anisotropic perturbation sampling rather than using isotropic distributions.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary benefit of CAZO for developers?

It significantly lowers CUDA memory usage, which is essential for deploying and adapting models on memory-constrained hardware.

Q2. Does this method require training the entire model?

No, it keeps the pretrained model weights frozen and optimizes only a set of minimal adapter parameters.

Q3. How does CAZO perform compared to existing methods?

It achieves high accuracy on benchmarks like ImageNet-C while utilizing significantly less memory than traditional techniques like TENT.

Q4. How does the method approximate gradients?

It uses zeroth-order random gradient estimation by sampling from a distribution to approximate gradients through function evaluations alone.

Q5. What is the role of the diagonal Hessian in CAZO?

It is used to construct a covariance matrix that enables anisotropic perturbation sampling, which helps address the variance issues found in standard zeroth-order methods.

Q6. How does the sliding exponential moving average function within this method?

It tracks the element-wise squared zeroth-order gradients to estimate the diagonal curvature of the loss surface.

Q7. What are the limitations of the current implementation?

Single-point perturbation results in degraded performance, so the method must use symmetric multi-point perturbation instead.

Q8. Which model architectures have been tested with this approach?

The authors tested ViT-B/16, DeiT, Swin-Tiny, and ResNet-50.

Q9. Does this method work for continual adaptation?

Yes, it achieved an average accuracy of 65.3 percent in continual test-time adaptation scenarios on ImageNet-C without model reset.

Browse the archive →
Benchmarks & Evals / Efficiency & Inference Benchmarking Triton Kernel Generation for AI The authors introduce RealisticTritonBench to evaluate LLM performance in generating production-grade Triton kernels for real-world AI frameworks. Efficiency & Inference / Benchmarks & Evals How Token Budget Changes LLM Rankings The paper demonstrates that model rankings shift significantly based on the token generation budget allowed during inference, challenging the reliability of standard static evaluation benchmarks. Benchmarks & Evals / Efficiency & Inference Optimizing Supply Chain Interventions Under Uncertainty The paper introduces a decision-aware ranking method for supply chains that selects interventions by balancing recovered net value against operational costs.