Back to Feed
Multimodal / Training & Fine-Tuning

Recognizing Mediterranean Dishes with OliveGemma

Original: OliveGemma: A 3 Billion Visual Language Model for Recognising the Mediterranean & European Diet
By Dimitrios I. Zaridis, Traianos Tsiokris, Vasileios C. Pezoulas, Daphni Plati, Eugenia Mylona, Eleni Georga, Nikos Tsiknakis, Antonis Sakellarios, Dimitrios I. Fotiadis Published 2026-08-04

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • Achieved a 92.96 percent top-1 accuracy in identifying complex plated Mediterranean meals.
  • Significantly outperformed frontier zero-shot models including GPT-5.4 Mini and Claude Haiku 4.6 by margins of 18 to 64 percent.
  • Utilized a custom dataset of over 102,000 instruction-style question-answer pairs derived from knowledge distillation.
  • Implemented a parameter-efficient fine-tuning approach using LoRA to adapt the base PaliGemma-2-3B architecture.

Summary & Methodology Analysis

The researchers developed OliveGemma to address the specific failure modes of general-purpose visual language models when classifying the nuanced, compositional nature of Mediterranean cuisine. They chose the PaliGemma-2-3B architecture as the foundation, which processes visual inputs through a SigLIP vision encoder, a component that maps image patches into a vector space shared with text embeddings. To optimize for a constrained 216-class vocabulary, they applied LoRA, or Low-Rank Adaptation, which freezes the pre-trained weights and injects small, trainable rank-decomposition matrices into the decoder layers to reduce computational overhead. The team trained the model across seven projection layers using a rank of 16 and an alpha of 32.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary purpose of OliveGemma?

It is a specialized visual language model designed to accurately identify Mediterranean and European dishes in images.

Q2. Why use OliveGemma instead of general-purpose AI models?

General-purpose models often struggle with the fine-grained, compositional nature of regional cuisines, whereas OliveGemma outperformed them by up to 64 percent in this domain.

Q3. What kind of data was used to train the model?

The team harmonized the MedGR, ODIN, and VIPPSTAR datasets and generated 102,642 instruction-style question-answer pairs using distilled knowledge from Anthropic Claude 4.6 Opus.

Q4. How was the training process structured?

It was a two-stage process where the SigLIP vision encoder was initially frozen for 1,500 steps, followed by fine-tuning with a reduced learning rate while keeping projection layers and embeddings frozen.

Q5. Which specific architectures were used for comparison?

The study compared OliveGemma against Gemini Flash 3, Gemini Flash 3.5, GPT-5.4 Mini, Claude Haiku 4.6, and various CNN baselines like ResNet-50, EfficientNet-B3, MobileNet-V3-Large, and DenseNet-121.

Q6. Does the model face any specific performance limitations?

Yes, while top-1 accuracy is high, the model's top-5 accuracy is marginally lower than the performance of traditional CNN baselines.

Q7. What is the parameter size of the base model?

The base architecture, PaliGemma-2, has 3 billion parameters.

Q8. Are there any specific privacy or reproducibility requirements addressed?

The paper aims to provide an accurate, privacy-preserving, and reproducible alternative to proprietary zero-shot vision models.

Q9. What does the paper specify regarding inference latency or hardware costs?

The paper does not specify hardware requirements, exact inference latency, or dollar-cost-per-request metrics.