Unified Lexical and Semantic SQLite Search

Summary & Methodology Analysis

The scrydb library provides a unified interface for document retrieval that leverages existing SQLite extensions to perform both traditional keyword-based lexical search and vector-based semantic retrieval. By implementing document binarization via a Heaviside step quantizer, which maps continuous vector values to binary bits, and scalar quantization to int8, which represents embedding weights using 8-bit integers, the system significantly reduces the computational overhead typically required for exhaustive searching. These techniques allow for efficient cross-precision retrieval, where the engine retrieves coarse-grained results first and reranks them using higher-precision embeddings. To combine these approaches, the library implements Reciprocal Rank Fusion, a technique for merging multiple search result lists into a single ranked output, ensuring that lexical and semantic signals are integrated directly within the database file.

Performance evaluations on the BEIR datasets demonstrate that these quantization configurations are highly effective. Specifically, the Hamming + cos int8 method achieves a top-10 ranking effectiveness comparable to exhaustive cos int8 search while operating at roughly one-fifth of the mean latency, specifically 164.5 ms versus 822.5 ms. Furthermore, scrydb proves competitive against the MTEB baseline, which uses the Qwen3-Embedding-8B model for document representation. For example, the Hamming + cos int8 configuration reaches 0.649 nDCG@10 on FiQA compared to the 0.646 baseline, while other configurations performed similarly or better on the Quora, SciFact, and Touché datasets.

Despite these gains, the architecture is not a replacement for purpose-built vector databases, such as those relying on HNSW graph-based indexing or inverted-file systems used in tools like Faiss. The paper emphasizes that scrydb does not address requirements for concurrent multi-tenant serving, horizontal sharding, replication across machines, or real-time ingestion for corpora that grow without bound. Because the system relies on exhaustive scanning, it remains impractical for workloads involving tens of millions to billions of documents on commodity hardware. The authors provide prebuilt databases including Qwen3-Embedding-8B embeddings to facilitate testing across the evaluated benchmarks.