Adding Syntactic Structure to Language Models

Summary & Methodology Analysis

This paper addresses a core limitation in standard Transformers, where the lack of explicit syntactic structure hampers the model's ability to handle compositional generalization. The proposed method, SiPE, injects syntactic inductive biases by utilizing a Hexatagger dependency parser to generate terminal and non-terminal tags for input sequences. These tags are mapped to learned embedding lookup tables, which are then integrated directly into the input embeddings for encoders or multiplicatively into the relative positional terms for decoders. To ensure the model effectively utilizes this structural information, the authors implement an auxiliary indicator-prediction objective that runs alongside the standard language modeling loss during training. This allows the model to learn syntactic priors without requiring changes to the existing self-attention architecture. Testing across various architectures, including RoBERTa, DeBERTa-v3, ModernBERT, and Transformer-XL, demonstrates measurable gains, such as an 8.2 percent improvement on the GLUE benchmark for Transformer-XL. However, the operational impact is significant, as the reliance on an external dependency parser at inference time adds a processing step. Additionally, the method is incompatible with standard KV-caching in decoder models because the tags may be subject to revision during the sequence generation process, which complicates standard optimization paths for generative inference.