Comparing Multimodal Pipelines for Document Retrieval

Summary & Methodology Analysis

This study evaluates document retrieval performance using the REAL-MM-RAG benchmark to compare textual, visual-single, and visual-multi pipeline architectures. Text-based pipelines rely on OCR to parse documents, which frequently leads to the loss of structural information, such as table headers or financial years, causing retrievers to return irrelevant results. In contrast, visual pipelines process full-page images directly, allowing them to capture layout-sensitive data that textual models miss. The authors demonstrate that even visual-single pipelines, which use a single dense vector representation, consistently outperform text-based methods across various metrics and query rephrasing levels.

The research compares two main architectural approaches for multimodal retrieval: dense models, represented by nomic-embed-multimodal-7b, and late interaction models, represented by ColQwen2.5-7b-multilingual-v1.0. Dense models map inputs to a single fixed-size vector. While efficient, these models struggle with rephrased queries because they depend heavily on surface-level term overlap. Late interaction architectures, which generate multiple sub-token or patch-level embeddings to allow for granular comparison between query and document, provide superior performance, especially when handling complex document layouts.

Despite these performance gains, there is a clear trade-off between effectiveness and resource consumption. Late interaction pipelines require significantly more storage for index sizes and introduce substantial latency. While text-multi pipelines maintain reasonable retrieval times, the visual-multi pipeline incurs a high retrieval latency that exceeds 20 seconds per query. These findings suggest that while multimodal vision-language models offer robust generalization, the selection of a pipeline must carefully weigh these operational costs against the specific accuracy requirements of the target domain.