Extracting Values From Bar Charts

Summary & Methodology Analysis

The Bar-JEPA method addresses the challenge of extracting numerical data from variable bar charts by first training a model on 100,000 synthetic samples. The system modifies an I-JEPA encoder, which is a joint-embedding predictive architecture that learns semantically rich features by predicting parts of an input from other parts, to support variable-resolution inputs using patch extraction techniques similar to Pix2Struct. This encoder is then frozen, and its latent features are passed to a decoder composed of deconvolution, batch normalization, and ReLU layers to generate heatmaps for identifying ticks, bars, and origin points. Detection coordinates are derived from these heatmaps using confidence-based non-maximum suppression (NMS), a technique that filters overlapping candidate detections to keep only the most confident one. Finally, numerical values are recovered by applying PaddleOCR for character recognition and RANSAC regression, an algorithm that identifies outliers while mapping pixel coordinates to numeric data. The results show that fine-tuning the encoder significantly improves performance, with a more than two-fold increase in F1 score over a vanilla encoder on synthetic data. On the UB PMC dataset, the model reached 45.0 percent accuracy at a 0.05 epsilon threshold and 34.1 percent accuracy at 0.02 epsilon. Despite these improvements, the method does not currently reach state-of-the-art results. The authors note that the architecture is limited to one specific type of chart and that integrating the frozen feature extractor into multimodal language models remains difficult.