recurrence and a novel relative positional encoding. The enhancements that were introduced in Transformer-XL help capture better long-term dependencies by attending to tokens from multiple previous segments segments. Our implementation is based on the codebase that was published by the authors of the Transformer-XL paper. Our implementation uses modified model architecture hyperparameters, our modifications recurrence and a novel relative positional encoding. The enhancements that were introduced in Transformer-XL help capture better long-term dependencies by attending to tokens from multiple previous segments