Quantization and Prompting in Biomedical LLMs

Summary & Methodology Analysis

The researchers evaluated three decoder-based models, specifically Mistral-7B-v0.3, BioMistral-7B, and Mistral-7B-Instruct-v0.3, using PubMedBERT as a supervised encoder baseline. They applied quantization, which is the process of reducing the precision of model weights to decrease memory usage, using both INT8 (LLM.int8) and INT4 (NF4) techniques via the bitsandbytes library. The evaluation focused on the PubMed RCT sentence classification task, employing custom answer-text prompt templates with both sum and mean-token normalization for scoring log-likelihoods. To handle probability distribution, they applied post-hoc temperature scaling, a technique used to calibrate model confidence scores after the initial training phase. Results indicate that specialized models retain high performance under INT8 compression, showing accuracy and F1 score deviations within 1 to 2 percentage points of FP16 baselines. The methodology also featured an evolutionary prompt optimization process, which iteratively searches for the most effective prompt structures to maximize classification accuracy. Limitations include the reliance on the PubMed RCT dataset, which may hinder generalizability, and the use of bitsandbytes as the sole quantization framework. Furthermore, the evolutionary prompt selection process suffered from potential leakage because the development subset was derived directly from the test set without a clean held-out validation procedure.